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1. Introduction and Background 


Groupware is a term describing an emerging computer software technology enhancing the 
ability of people to work together as a group, (a software driven "group support system"). 
This project originated at the beginning of 1992 and RICIS reports were issued describing 
the activity through May, 1995. These reports stressed the need for process as well as 
technology. That is, while the technology represented a computer assisted method for 
groups to work together, the Group Support Systems (GSS) technology also required an 
understanding of the facilitation process electronic meetings demand. Even people 
trained in traditional facilitation techniques did not necessarily painlessly adopt 
groupware techniques 1 . The latest phase of this activity attempted to 1) improve the 
facilitation process by developing training support for a portable groupware computer 
system, and 2) to explore settings and uses for the portable groupware system using 
different software, such as Lotus Notes. Below summarizes the groupware related RICIS 
activities. 

Phase I: June 1992 - May 1994 

> Established a research activity between NASA/JSC and RICIS to study how 
groupware software could improve productivity and stimulate team building (RICIS 
IR01.B, “Group Decision Support Requirements Review”— 6/1/92-5/31/94) 

> Completed survey of the field 

> Recommendations issued by a Project Management Team 

> Drafted requirements for a decision support and groupware facility housed at the 
University of Houston-Clear Lake 

> Accomplished establishment of a groupware meeting room at UHCL 

> Studied various methods of accomplishing electronic meetings 

> Adopted the “Interaction Method.” of process facilitation (The Interaction Method is 
based on the notion of shared responsibility as a result of a collaborative attitude, strategic 
thinking, and facilitative behavior.) 

> Held pilot sessions with NASA/JSC and UHCL groups 

> Issued a final report 

Phase II: June 1994 - May 1995 

> RICIS continued operating the UHCL groupware meeting room 

> Established a research activity with NASA/JSC studying Group Support Systems 
(RICIS IR01.C, “Group Support Systems”- 1/1/95-5/3 1/96) 

> Purchased a portable computer system capable of bringing the meeting to any room 
accomplished 

> Identified JSC personnel to participate in training sessions 

> Planned and held NASA focus group sessions 

> Presentations were given to upper-level managers 


Group Decision Support Requirements Review . (Gary Hamel, Charles Hardwick, 
Ravin Wijesinghe, June, 1994) and Group Support Systems June 1994 - May 1995 . 
(Gary Hamel, Charles Hardwick, Judy Solecki, March, 1996). 
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> Training classes held for JSC and UHCL personnel in conjunction with a groupware 
software vendor and a meeting process improvement consulting firm 

> Issued a final report 


Phase III: June 1995- May 1996 

> Moved the UHCL groupware facility from UHCL’s Bayou Building to UHCL’s Delta 
Building 

> Continued operating the UHCL groupware meeting room 

> Established a research activity and conducted JSC Lotus Notes surveys using the 
portable groupware computer system (IR13.A, “360 Degree Groupware Survey”- 
12/1/95-8/31/96)) 

> Produced a training support handbook for the portable GSS system 

> Developed a groupware and GroupSystems for Windows training aid consisting of 
thirteen computer disks and a script 

> Issued a final report 


2. The Lotus Notes Groupware Survey 2 

2. 1 Background, Objectives and Overview 

Employee performance review is a commonplace procedure in most organizations. 
Historically, a performance review consists of a supervisor evaluating a subordinate. 
More recently, though, survey methods to obtain a wider evaluation of an employee from 
co-workers and customers have emerged. These are called 360 degree performance 
reviews or 360-degree performance appraisals . 3 People at NASA were curious if 
groupware could enhance this peer review process. 

NASA/JSC had formerly conducted such evaluation processes using pen and paper. The 
process included distributing questionnaires, collecting completed forms, compiling data, 
and distributing the results. However, the user participation and results traditionally 
obtained had been low. 

NASA/JSC was looking for ways to identify issues among the teams working for the 
Space Station Program Office. To improve team performance and to identify the 
shortcomings among the members of teams, it was decided to anonymously exchange 
thoughts, ideas, and suggestions between the team members using groupware. 
Furthermore, use of groupware aimed to increase user participation to include non 
contractors in addition to the Space Station Program Office employees. The overall 
objective was to obtain constructive feedback to improve teamwork. 


Ravin Wijesinghe was the lead writer for this section of this report. 

See: http://deming.eng.clemson.edu/pub/tqmbbs/tools-techs/360pa.txt , 
Companies Evaluate Employees from all Perspectives , Jayart Kirksey, et. al., an article 
first published in Personnel Journal. 
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Consequently, a prototype survey session using Lotus Notes, a leading groupware 
software, was held in the Summer, 1995. During that time a 360 Degree Peer Evaluation 
was conducted at the NASA Johnson Space Center’s Space Station Program Office, 
assisted by the Human Resources Development Branch and the Research Institute for 
Computing and Information Systems (RICIS). The success of that event led to a desire to 
continue and expand the scope of the activity. 

The first survey activity held in the summer, 1995 was a “same place, different time” 
activity. That is, a networked portable computer system was set up in a room at JSC. 

The system used a portable server and ten laptop computers. NASA Civil Servants visited 
the room to complete a questionnaire to evaluate their team members. Although the 
participants of the survey had mixed reactions, the overall results of the feedback was 
overwhelming. The support for the teams were considered an essential element in 
aligning with organization goals, and supporting the peers to identify strengths and 
weaknesses. Performance feedback obtained were: 

• per team member 

• per team 

• per department/mail code 

A second 360 Degree Peer Evaluation was also conducted at the NASA Johnson Space 
Center’s Space Station Program Office, assisted by the Human Resources Development 
Branch and the Research Institute for Computing and Information Systems (RICIS) in 
Spring of 1996; user participation was increased to include customers, suppliers, 
members, and other associated contractors. 


3. System/Application Requirements 

3. 1 The Summer 1995 360 Degree Peer Evaluation Process 

3.1.1 Requirements 

The application requirements for the Summer 1995, 360 degree peer evaluation process 
included: 

• Easy to use 

• A combination of multiple choice, and open ended questions 

• Ability for a participant to rate a team member at a convenient time 

• Participants may not review his/her or others ratings 

• Anonymity 

• Ability for a participant to rate a person whom he/she works closely, but is not 
a team member of a common team 

• Ability for participant to keep track of people he or she rated 

• Obtain session feedback, when participant has completed evaluating 

• Provide a hard copy(paper) of summarized feedback results 

• per team member (to team member & team lead) 

• per team (to team lead & department head) 
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• per department/mail code as a whole (department head) 

• per program as a whole (department head) 

• ratings from others not in his/her team (per team member) 

3.1.2 Assumptions 

The following assumptions were identified and stated prior to developing the system. 

• Honor System 

• Participant may not rate him/herself 

• Participant may rate a team member once 

• Participant may not rate members of other teams 

• User base 

• Over 340 team members 

• 46 teams 

• 9 Department/mail codes and a category “Other” containing team 
member names 

3.2 The Spring 1996 360 Degree Peer Evaluation Process 
3.2.1 Requirements 

The application requirements for the Spring 1996, 360 degree peer evaluation process 
were: 

• Easy to use 

• A combination of Multiple Choice, [5 point scale], and Open Ended questions. 

• Ability for a participant to rate teams and team members at a convenient time 

• Participants may not review his/her or others ratings. 

• Anonymity 

• Include customers, suppliers, members, and other associated contractor types 
of rator’s 

• Separation of team and individual evaluations - (different questions based on 
rator type) 

• Ability for a NASA employee to switch between individual evaluations and 
team evaluations 

• Restricting others (non members) from accessing individual evaluation forms 
(Form access level security using first 6 digits of a person’s Social Security 
number) 

• Prevention of self evaluation for team members - (using first 6 digits of a 
person’s Social Security number 

• Ability for participant to keep track of people he or she rated 

• Obtain session feedback, when participant has completed evaluating 

• Ability to quickly summarize the evaluation forms 

• Provide a hard copy (paper) of summarized feedback results for individual 
evaluations 

• per team member (to team member & team lead) 
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per team (to team lead & department head) 

per department/mail code as a whole (department head) 

per program as a whole (department head) 

provide a hard copy (paper) of summarized feedback results for team 
evaluations 

per team (to team lead & department head) 

per department/mail code as a whole (department head) 

per program as a whole (department head) 


3.2.2 Assumptions 

The following assumptions were identified and stated prior to developing the system. 

• Honor System 

• Participant may not rate a team more than once. 

• User base 

• Over 340 team members 

• 46 teams 

• 8 Department/mail codes 

• 5 types of rator’s (Member of a team, Customers for a team. Suppliers 
for a team. Others (categorized as non-customers, non-suppliers). Peer 
to a team member) 

4. Application Development 

GroupSystems for Windows, GroupSystems V, and Lotus Notes were the groupware 
software packages considered for use in the peer review evaluation process. 

4. 1 GroupSystems for Windows and GroupSystems V 

GroupSystems for Windows was not yet incorporated into the RICIS GSS environment 
when the peer evaluation project was initiated. GroupSystems V, a DOS based 
application that supports a distributed environment, was considered as a tool to use for 
the survey. GroupSystems V’s survey tool has the capability of obtaining different types 
of information, e.g., text, multiple choice etc. For example, a single form may be 
configured to rank order a list of items, select the most suitable item from a list of items 
(multiple choice), and to obtain open ended answers. The results may be viewed with 
statistical data which offers a method of identifying consensus 4 . 


However, GroupSystems for Windows and GroupSystems V each require a technical 
facilitator to enable users to access the tools. The first electronic peer evaluation process 
included two separate questionnaires for two different groups of users (team members 
. and contractors). Using either of the versions of Group Systems software, a technical 
facilitator would have to start each participant depending on the user category. 

Evaluating multiple peers would require the facilitator to start sessions multiple times for 

4 See Group Decision Support Requirements Review . (Gary Hamel, Charles 
Hardwick, Ravin Wijesinghe, June, 1994) for a review of these tools. 
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each rator. To accommodate over 340 rators rating multiple peers would not have been 
practical nor possible within a short period of time (three weeks). In addition, the 
requirements to summarize data separated by Team Member Name, Team Name, 
Department, and Program is not supported by either of these software packages. 

After review, it was decided that GroupSystems could not support the demands stated in 
the “requirements” section above. The systems needed to be customized using a 
distributed environment supporting, secure, easy to use, programmable groupware. 


4.2 Lotus Notes 

Lotus Notes proved to be an ideal platform to develop the peer evaluation system since it 
provided the interface to customize and develop an application according to a set of 
requirements. Information may be gathered as documents using Forms, and a collection 
of documents may be summarized and reviewed using Forms or Views (A View 
summarizes the information in a document in a tabular format). 

During the design and development phases, new requirements can be easily included 
without affecting the data or the rest of the design. Input and output screens can be 
customized and changed as easily as modifying text using a word processor. The 
collection of data can be summarized by selecting documents based on search patterns 
and Forms used. For example, summary of the survey results for “Team X” as evaluated 
by member of “Team X”, or , summary of the survey results for “Team X” as evaluated 
by “Customer Y”. In addition to its distributed accessibility, Lotus Notes also provides 
security for accessing databases. Forms, Views, documents, fields within a Form, and 
sections within a Form. 

However, Lotus Notes does not offer statistical functions to summarize the data within a 
Form. Statistical data may be viewed in a tabular format using a View. Therefore, to 
include statistical information within a form summarizing the documents requires fields 
to be programmed to calculate such statistical requirements such as mean, standard 
deviation, etc. (See Appendix, A-3b, for a review of the Summer 1995 Peer Evaluation 
results; see Appendix B-2b, B-4b, B-5b, B-6b, and B-7b to see the summary of the 
session evaluations of the Spring 1996 Peer Evaluation Survey) 

5. General Description of the Process 

The first peer evaluation conducted in the Summer, 1995 was located at a critical 
geographical location near the elevator to the Space Station Program Office. The 
participants were restricted to NASA Civil Servants. Process steps required were as 
follows: 

• Participants may walk into the evaluation room and review a list of team names (See 
Appendix A-2) 

• Select team name and obtain list of team members 

• Select member and obtain an evaluation form to rate. (See Appendix 
A-3a) 
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• Rate/answer all questions (open ended was optional) and submit the 
evaluation 

• Select next member and complete evaluation 

• When done with the team, repeat process for other teams 

• When survey is completed, provide session evaluation prior to leaving (See Appendix 
A-4a) 

The second phase expanded upon the initial activity in several ways. One significant 
difference was in the number of participants and the location of the participants. This 
survey included NASA Civil Servants as well as NASA contracted employees, i.e., 
Members, Customers, Suppliers, and Others. (See Appendix B-3) The participants 
resided at the Johnson Space Center and NASA facilities in Washington DC, Alabama 
and Florida. Nearly 350 NASA Civil Servants and approximately 650 contractor 
personnel were expected to participate. The same portable computer system that was 
utilized in the previous survey activity was setup to obtain the evaluations. The location 
of the system remained in the same room at NASA/JSC. The process steps were 
expanded as follows: 

• Participants may walk into the evaluation room and request for list of team member 
names (See Appendix B-l) or team names. (See Appendix B-3) 

If the rator is evaluating a team as a Member, Customer, Supplier, or Other, 

• Select team name and request for an evaluation form 

• Identify relationship to the team (See Appendix B-3) 

• If Customer, Supplier or Other, identify organization or team of rator 

• Rate/answer all questions (open ended was optional) to the appropriate 
form given by system, and submit the evaluation (See Appendix B-4a, 
B-5a, B-6a, B-7a) 

• Select next team name and repeat evaluation process for other teams (If rator 
relationship changes, obtain new relationship and identify organization or 
team of rator) 

If the rator is evaluating an individual in his/her team, 

• Provide Identification by entering the first six digits of the Social Security 
number 

• Select a team in which the rator is a member and obtain list of team members 
(rator should not be able to rate teams in which he/she is not a member) 

• Select member and obtain an evaluation form to rate (rator should not 
be able to rate him/her self) (See Appendix B-2a) 

• Rate/answer all questions (open ended was optional) and submit the 
evaluation 

• Select next member and complete evaluation 

• When done with the team, repeat process for other teams in which rator is a 
member 

• Rator may switch between team evaluation and individual evaluation. 
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• When survey is completed, provide session evaluation prior to leaving. (See 
Appendix B-8a) 

The schedule for system design, development and implementation of second peer 
evaluation schedule is shown in Appendix C). 

5. 1 Development Platform 

Lotus Notes database can contain one gigabyte of information in a single database. 
Therefore, the size of data generated, ability to do continuous assessment without creating 
user id's for all participants, manipulation of data according to NASA requirements, not 
requiring a technical facilitator to enable access to the required forms, possibility to 
generate/obtain a form for a selected team or individual were factors supporting the use of 
Lotus Notes as an application development platform. 

5.2 Summer 1995 360 Degree Peer Evaluation Process 

5.2.1 System Input 

To initialize the system, setup data containing Member Name, Department/mail code, and 
Team Name were obtained in an EXCEL spreadsheet in column format. The data were 
directly imported to the Lotus Notes Database. 

A total of 8 questions (6 multiple choice, 2 open ended) were formulated. (See Appendix 
A-3a) 

5.2.2 Process Steps (Summer ‘95) 

The process steps/directions (See Appendix A-l) were defined as follows: 

• Use the arrow keys to highlight department/mail code 

• Press an icon (team icon)' to view team names 

• Use the arrow keys to highlight teams’ name and press an icon to (team icon) 
view team members 

• To begin rating a team member, use the arrow keys to highlight a name and 
press an icon (compose icon) to compose an evaluation form 

• Highlight the selected answer to the multiple choice questions and press 
<enter> or left mouse button to select 

A user may not abstain from answering any of the multiple choice questions, 
and should not be able to proceed without a choice 

• Type response to open-ended questions 

Prompt user if open ended questions are not answered, but may submit only 
the ratings for the multiple choice questions 

• Once survey evaluation form has been filled, submit ratings by pressing an 
icon (mail icon) 

• Highlight next team member and repeat process 

• Repeat process for other teams in which the rator/participant is listed in 

• Evaluate persons in other teams whom evaluator works closely with, but is not 
listed as a team member, in department category listed as “OTHER” 
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• If team or department is unknown, a participant may locate a person by typing 
the first name, last name or both in the search box 

• Once completely finished rating, press walk-away icon to initialize the screen 
and terminal, and to obtain the session evaluation form 

5.2.3 Output Requirements 

• Feedback data were to be summarized and distributed 

• per team member (to team member & team lead) 

• per team (to team lead & department head) 

• per department/mail code as a whole (department head) 

• per program as a whole (department head) 

• per ratings from others not in his/her team (per team member) 

5.2.4 Forms Developed 

The “Team Initialization Form” was created to import data from an Excel spread sheet. 
The form contained fields such as Member Name, Department/mail code, and Team 
Name. Names which were not entered/imported from the Excel spreadsheet could be 
easily entered to the system at time of survey. 

The “Evaluation Form” contained the questions defined for the survey. The form 
recorded information such as the team name, department/mail code, the name of the ratee, 
(if the evaluation was for an individual). In addition, field entries were validated to 
obtain correct information. The open ended questions were optional. The information 
recorded from the Evaluation form were maintained in a separate view for summarizing 
the information for Output Requirements. 

“Feedback Summary Forms” (Summer ‘95) were created to summarize session 
responses for distribution among the team members, team leads, and the program 
manager. In these feedback summary forms: 

• Multiple choice questions were rated numerically in relation to its position in 
the list of choices. (Option 5 was disregarded for evaluation, See Appendix 
A-3a) 

• Statistical information such as number of number of responses for question, 
mean, standard deviation were calculated 

• A descriptive analysis of ratee with respect to the results (mean) was 
formulated 

• Open-ended suggestions/ratings were appended/merged to form a collection of 
comments 

The output requirements included summarizing to provide statistical information, and 
collecting the open ended answers in paragraph form. (See Appendix A-3b) In addition, 
the multiple choice questions were programmed to analyze the results, and to provide a 
general description of the overall result. Thus, a form (Form to summarize data per 
department/mail code) was created to summarize the information by department/mail 
code. To provide feedback on team performance, all surveys related to each team were 
summarized using the form summarizing data per team. To provide feedback on 
individual performance, all surveys related to each individual were summarized using a 
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form summarizing data per team member from peers. To provide feedback on 
individual performance as evaluated by non team members (others), all surveys related to 
each individual were summarized using a form summarizing data per team member 
from others. A session evaluation form was created to obtain an overall review of the 
survey process. To review the session evaluation form and the results of the session 
evaluation, refer to Appendix A-4a and A-4b. 

5.2.5 Views Developed 

The view “List of department/mail codes” was created to list the organizations teams, 
and team members sorted by the department/mail code. The view was created as the 
defaults view seen by the rators, and to obtain feed back summary, per department. For 
example, select the department from the list of departments and compose the form “Form 
to summarize data per department/mail code”. The view “List of teams” was created to 
list the organizations teams, and team members sorted by the team name. The view was 
created to obtain feed back summary per team. For example, select the team name from 
the list of teams and compose the form “Form to summarize data per team”. The view 
“List of team members” was created to list the organizations team members sorted by 
the team member name. The view was created to obtain feed back summary per team 
member from peers, and per team member from others. For example, select the team 
name from the list of teams and compose the form “Form to summarize data per team 
member from peers” or “Form to summarize data per team member from others”. 

5.2.6 Description of System 

To automate the processes and functionality of the system, five icons were programmed 
to execute macros and validate entries. (See Appendix A-2) In addition, the system was 
developed to change the names of the individuals already rated by the rator. Thus, the 
change in color is a prompt for the rator to identify the individuals or teams already 
evaluated. The walk away icon was created to automatically obtain a session evaluation 
form at time of survey completion. 

5.3 Spring 1996 360 Degree Peer Evaluation Process 

5.3.1 System Input 

Building upon the experienced based on the ‘95 survey, the system input remained the 
same except for an additional field which was the Social Security number used to fulfill 
the security requirements stated in the above “Requirements Spring ‘96” section. A total 
of five separate questionnaires were formulated to obtain team responses from members, 
customers, suppliers, others, and individual responses from team members. (See 
Appendix B-2a, B-4a, B-5a, B-6a, B-7a) The member form contained 13 questions (11 
multiple choice, 2 open ended), the customer form contained 10 questions (8 multiple 
choice, 2 open ended), the supplier form contained 10 questions (8 multiple choice, 2 
open ended), the other form contained 10 questions (8 multiple choice, 2 open ended), 
and the individual form contained 1 1 questions (9 multiple choice, 2 open ended). 

5.3.2 Process Steps 

The process steps/directions were defined as follows: 
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For Individual Evaluations, 

• Press an icon (individual icon) and enter first six digits of their Social Security 
number to view department names 

• Use the arrow keys to highlight department/mail code 

• Press an icon (team icon) to view team member names 

• To begin rating a team member, use the arrow keys to highlight a name and 
press an icon (compose icon) to compose an evaluation form 

• Highlight the selected answer to the multiple choice questions and press 
<enter> or left mouse button to select 

A user may not abstain from answering any of the multiple choice questions, 
and should not be able to proceed without making a choice 

• Type response to open-ended questions 

Prompt user if open ended questions are not answered, but may submit only 
the ratings for the multiple choice questions 

• Once survey evaluation form has been filled, submit ratings by pressing an 
icon (mail icon) 

• Highlight next team member and repeat process 

• Repeat process for other teams in which the rator/participant is listed in 
For team evaluations, 

• Press an icon (team icon) and enter first six digits of your Social Security 
number to view department names 

• Use the arrow keys to highlight department/mail code 

• Press <enter> or left mouse button view team names 

• Use the arrow keys to highlight teams’ name and press an icon to (team icon) 
view team members 

• Identify relationship to team (member, customer, supplier, or other), and 
identify the rator team name 

• To begin rating a team member, use the arrow keys to highlight a name and 
press an icon (compose icon) to compose an evaluation form 

• Highlight the selected answer to the multiple choice questions and press 
<enter> or left mouse button to select 

A user may not abstain from answering any of the multiple choice questions, 
and should not be able to proceed without a choice 

• Type response to open-ended questions 

Prompt user if open ended questions are not answered, but may submit only 
the ratings for the multiple choice questions 

• Once survey evaluation form has been filled, submit ratings by pressing an 
icon (mail icon) 

• Highlight next team member and repeat process 

• If team or department is unknown, a participant may locate a person by typing 
the first name, last name or both in the search box 

• Once completely finished rating, press walk-away icon to initialize the screen 
and terminal, and to obtain the session evaluation form 
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5.3.3 Output Requirements 

• Feedback data were to be summarized and distributed (See Appendix B-2b, B- 
4b, B-5b, B-6b and B-7b) 

• per team member (to team member & team lead), as evaluated by 
immediate team members 

• per team (to team lead & department head), as evaluated by members, 
customers, suppliers, and others 

• per department/mail code as a whole (department head), as evaluated 
by members, customers, suppliers, and others 

• per program as a whole (department head), as evaluated by members, 
customers, suppliers, and others 

Session feedback was obtained from the perspective of an individual evaluator and a team 
evaluator. (See Appendix B-8a, B-8b, B-8c for the session evaluation form and the 
session feedback results.) 

5.3.4 Forms Developed 

The “Team Initialization Form” contained fields such as Department/mail code, Team 
Name, and Team Leader(s). Records which were not entered/imported from the Excel 
spreadsheet could be easily entered to the system at time of survey. 

“Rator Initialization Form” contained Department Name, Ratee Name, and Employee 
ID (first six digits of Social Security number). 

The five “Evaluation Forms” contained the questions defined for the survey (See 
Appendix B-2a, B-4a, B-5a, B-6a, and B-7a). The form recorded information such as the 
team name, department/mail code, the name of the ratee (if the evaluation was for an 
individual). In addition, field entries were validated to obtain correct information. The 
open ended questions were optional. The information recorded from the evaluation form 
were maintained in a separate views for summarizing the information for output 
requirements. The type of information summarized in the “Feedback Summary Forms 
(Spring ‘96)” remained similar to the “Feedback Summary Forms (Summer ‘95)”. 

5.3.5 Views Developed 

In addition to the views developed for the peer evaluation for Summer, 1996, additional 
views were created to list team information for team evaluations, responses by members, 
responses by customers, responses by suppliers, and responses by others. 

5.3.6 Description of System 

The system description remained similar to the Summer 1996 survey. However, 
additional security was added to prevent repetitive evaluations, self evaluations, and to 
prevent non team members evaluating individuals. 

5.4 Anticipating Disaster Recovery 

In anticipating disasters, a system data was backed up on a daily basis. In anticipation of 
natural disasters, the backup tape was physically located in a separate location. 
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6. Implementation/Survey 

6.1 Implementation Phase (August 1995) 

• Hardware 

• 10 user terminals (laptops) & 1 server (laptop) 

• Configured to provide privacy 

• Configured and setup in a room close to the elevator, and is visible and 
easily accessible by all participants 

• Adequate posters on the walls of the computer/evaluation room 
prompting user steps and describing screen layouts 

• A written sign-in sheet to track system usage and attendance 

• Tape drive for daily backups 

• HP Laser Printer 

• Photo copy machine 

• Paper, envelopes, paper clips, labels, etc. 

• Software 

• Lotus Notes Release 3.0 

• Windows 3.11 (Windows for Workgroups) 

7. Distribution of Results 

The printed summary of results were distributed to each individual, team lead, and 
program manger. Individual results were distributed to the individual and the team lead. 
The team results were distributed to the team leads and the program manager. The 
department results were given to the program manager. 

8. Lessons Learned 

8. 1 Analysis of Participants Comments and Feedback 

8.1.1 Session Evaluation (Summer 1995) 

• % User participation - 100% 

• % Users who thought of the survey as a good way 

to collect team ratings - 69.73% 

• % Users who did not think of the survey as a good way 

to collect team ratings - 25.85% 

• . % Users who abstained from the session evaluation 

4.42%; 

The suggestions for improving the rating process obtained from user participants is 
attached to Appendix A-4b. 

8.1.2 Session Evaluation (Spring 1996) 

The session evaluation questions for Spring 1996 were changed to obtain feedback for the 
individual evaluation process, and the team evaluation process. The results are attached 
to Appendix B-8b and B-8c. 
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8.1.3 Organizational Recognition Award 

On February 24, 1995, Ravin Wijesinghe, a graduate student Research Assistant for the 
Research Institute for Computing and Information Systems at the University of Houston- 
Clear Lake received a Certificate of Appreciation from the Space Station Program Office 
at the Program Managers Office, Houston, Texas. 

The award was presented by Mr. Randy Brinkley, the Space Station Program Manager, in 
appreciation of the excellent computer and customer support of the Space Station 
Program Office peer evaluation pilot program provided by Mr. Wijesinghe and the RICIS 
staff. 

9. Group Support Systems Operational Support 


The above section summarized the successful implementation of Lotus Notes groupware 
for a Peer Evaluation process under RICIS activity number IR13.A. The following 
reports on the implementation of groupware training through RICIS research activity 
IR01.C. 

Phase Two (see opening “Phases” chart) of the effort to introduce electronic meeting 
technology brought together a representative group of NASA/JSC people from assorted 
organizations within NASA/JSC. 5 That group went through two facilitation training 
workshops. The assumption was that familiarity with groupware technology would help 
the participants educate their managers about electronic meetings and that management 
would support continued development of groupware training and practice sessions. 

RICIS staff supported the continued training in several ways. First, RICIS staff were 
available to provide training and facilitation assistance to JSC as needed, based upon the 
budget allocation of the project. Second, RICIS staff produced a training support 
handbook for the portable GSS system. The handbook provides basic suggestions for set- 
up and solutions in case of technical glitches using the portable system. (See Appendix 
One) 

Third, RICIS staff also worked on producing other training aids to assist JSC personnel 
with electronic meeting operations. The original plan was, with HRD assistance, to 
develop a short (3-5 minute) video to be played at the beginning of each practice session. 


This group was called a Virtual team" in the previous reports; the people in the 
team assembled for groupware training and then returned to their respective organizations. 
Virtual teams assemble from various parts of an organization as tasks require. They draw 
from people across traditional organizational boundaries and dissolve as the task is 
completed. Traditional staffing pulls teams from within a department or unit. See Group 
Support Systems June 1994 - May 1995, (Gary Hamel, Charles Hardwick, Judy Solecki, 
March, 1996) for details about the JSC virtual team. 
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The video was to describe groupware and the tools/techniques available for the practice 
sessions. In late 1995 it was decided, after discussion with the HRD representative for 
this project, to create a computer driven program instead. This decision was made to take 
further advantage of the portability features of the portable groupware system. The 
training modules that replaced the video could be plugged into the overhead data 
projector of the portable computer system at the start of a session. After a brief 
introduction to groupware as a concept, the facilitator would then highlight the tools that 
were selected for use at the meeting. Participants would see what their computer screens 
would look like and be given a quick briefing on how GroupSystems for Windows 
functioned. This training aid consists of thirteen computer disks and a script. (See 
Appendix Two). 


10. Beyond the Technology 

As noted earlier, the portable GSS system was successfully used to explore groupware 
applications using Lotus Notes. However, due to license, logistic and organizational 
issues and forces, JSC use of the portable GSS system and the GSS meeting room at 
UHCL was minimal; the virtual team did not reassemble and no practice sessions 
occurred. What impediments caused the project to stall? 

One stumbling block slowing progress in this project was the move of the GSS lab at 
UHCL from one building to another. The UHCL groupware room was housed in the 
Bayou building on that campus. The university moved the GSS lab to a new building 
over the Summer of 1995. In effect, the move shut down the lab from July through 
September 21, 1995. This interrupted the continuity of the setting and it took time to get 
the UHCL GSS system up and running properly. Perhaps more importantly, the system 
was "out of sight and thus out of mind" and unavailable for a length of time, making it 
easy to leave it out of future planned activities. 

Another impediment was the lack of a software license for the portable system. Ventana 
released a Windows version of the software that was much more user friendly than their 
old GroupSystems V DOS version. License agreements were in place through UHCL to 
cover use of the software and the extra user stations provided by adding the portable 
computers in the RICIS GSS room. However, no budget had been allocated to purchase 
the Window's version for the portable GSS system server. RICIS asked a Ventana 
representative that Ventana grant a $250 research license for use on the portable system to 
help understand issues involving portable GSS use. Despite several phone contacts with 
Ventana representatives, no resolution as to how to load a research license of the software 
on the portable GSS server was reached. Without a budget for the GroupSystems 
software the portable system could not be used as a stand alone groupware system. 


Understanding the logistics of the portable GSS system and the shared use of it by two 
organizations also was a factor. The cooperative agreement between UHCL and 
NASA/JSC that flows through RICIS was designed so that both institutions share the 
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research and the equipment. The portable system was stored at JSC and initially there 
was confusion about the procedure for RICIS personnel to gain access to the system. 

Once clarified, security badging and physical access to the Johnson Space Center and to 
the office where the portable system was stored proved inconvenient for RICIS personnel 
and hindered the initiative of RICIS personnel, especially the non-U.S. citizen student 
assistants. In retrospect, it would likely have been better to house the portable system at 
RICIS. In that way the system would have been used by staff and students to practice and 
JSC would be "on call" i.e., whenever the system was needed JSC could simply phone 
RICIS and made arrangements for RICIS staff and students to go to JSC and set up the 
system for a session. 6 

Organizational issues and downsizing pressures at both UHCL and JSC added further 
complications. The project's Human Resources Directorate (HRD) contact was not 
available between February and mid May, 1996. No calls for training assistance from the 
HRD office were logged during this phase of the project, and RICIS did not have an 
active named contact to communicate with during this time frame. This lack of 
communication helped keep the practice training sessions at a standstill. The deliverables 
in the contract called for a matrix outline of meeting types to be developed by RICIS 
based upon data provided by the HRD office. The plan was to take the meeting type 
scenarios and establish practice sessions to allow continued technical facilitation training. 
The schedule of training sessions was to be coordinated between RICIS and Human 
Resources Development office. The data for the matrix of JSC meeting types was not 
delivered from JSC until the end of May 1996. (See Appendix Three) Lack of progress 
on this front seriously weakened the model of operation and hindered reassembly of the 
JSC virtual team. 


11. Final Observations 

Introducing new information technology into today's turbulent environments proves 
vexing. Groupware software is not merely a "plug and play" technology. The technology 
requires skilled facilitation to function seamlessly. Beyond the technology and facilitation 
skills lies a need for commitment from organizations and managers in those 
organizations. Neither the technology nor a few dedicated advocates of the technology 
will make much progress in the face of organizational inertia. 

In retrospect, a fundamental aspect of this research activity involved using GSS 
technology as a change agent within organizations. While the generally favorable 
reviews of GSS session participants remained constant, (See Appendix Four) the funding 
support from the organizations they represented failed to materialize at any meaningful 
Operational level. A detailed study of why GSS participants favorably raved while those 


A hand truck style cart to transport the portable computer system between 
buildings was to be obtained, as outlined in the research activity description. As portable 
GSS system use was minimal, the cart was not purchased. 
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with the purse strings kept them closed would shed insight into how, and if, future 
implementation of GSS technology should proceed. Two sample study questions are 
addressed below. 

First, should the GSS project have been cast as a reengineering effort? For example, a 
case study of information technology reengineering at Amoco Production Company listed 
three components for successful renewal, expressed through this formula: 

D xVxT> Resistance to Change. 

D stands for the dissatisfaction with the "current state " 

V stands for a vision of a desirable future 
T stands for the tools to get there. 

The Amoco study pointed out that "[a]ll three factors need to be clear and in place. The 
omission of any one (a zero in the formula) nullifies the left side of the equation, and 
change will not occur." 7 

Did the GSS tool technology sing a siren song for those who conceived of this research 
activity and embraced the technology; ("build a GSS lab and they will come")? Previous 
reports from this research activity pointed out that introducing groupware is an extensive 
procedure requiring a model of operation and skilled facilitation. In this perspective, 
placing that process in an organizational vision with top management commitment 
proved impossible during this research activity, both at UHCL and at NASA/JSC. 

Second, perhaps the transfer of the groupware technology was not adequately embraced 
by the organizations because not enough time was allocated to allow the organizations to 
adopt GSS? One widely-used explanation of technology adoption breaks people into a 
bell-shaped curve. At the far left of the curve are "opinion leaders". This RICIS/NASA 
GSS research activity was sparked by a few forward looking NASA employees who saw 
the technology as a means to improve meeting efficiency. The resulting research activity 
built a GSS lab at UHCL and provided training sessions for NASA and UHCL 
employees. A portable GSS system was also procured but not widely utilized. According 
to the model, the GSS project never went much beyond the opinion leader stage, though a 
few UHCL professors, especially in the School of Business, became "early adopters," the 
next stage of acceptance of new technology in this model. The model's remaining 
categories shows early adopters followed by "early majority," "late majority," "laggards," 
and "retards". 8 Bringing groupware into the early majority stage of this model in these 


"Information Technology Reengineering at Amoco Production Company," Jack 
R.Camp, Amoco Exploration and Production Technology, Houston, TX., Information 
Systems Research Center Seminar, University of Houston, 4/18/96 
8 "The Continuing Impact of Information Technology on Business and Commerce" 
A.M. Jenkins, Center on the Management of Information Technology Seminar, 5/2/96, 
RICE University, Houston. “The Rate of Adoption” graph is from the session handout. 
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organizations lies in the future; it depends upon the vision of the organizations to adopt 
information technology to improve process and team performance. 


RATE OF ADOPTION 



i 

OPINION 

3 

EARLY 

*5 

EARLY 

7 

LATE 

9 

LAGGARDS RETARDS 

LEADERS 

ADOPTERS 

MAJORITY 

MAJORITY 



For now, neither NASA nor UHCL has allocated sufficient funding to meaningfully 
continue this research activity; RICIS has laid off the professional GSS support staff. 
Through RICIS, UHCL and JSC assembled a world class groupware facility. The vision 
and support for robust continual and future operation has not materialized. As evidenced 
by the Internet, the power of networked computer systems is increasing. Drawing upon 
these networks, groupware will likely play a significant part in the collaboration among 
teams in successful organizations of the future. 
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APPENDIX A 
Summer 1995 



APPENDIX A-l 
Directions and Helpful Hints 



Directions 



1. Use the arrow keys to highlight the mail code where your team is located. 


2. Press the “expand” icon 



to view team names. 


3. Use the arrow key to highlight your teams’ name, and press 
team members. 


to view 


4. To begin rating your team me mbers, use the arrow keys to highlight a name 
and press the “compose” icon 1 31 . 

5. Read each question and use the arrow keys to move the underline to the response 
you wish to make. The <enter> key will mark your answer. You can then use the 
arrow keys to move to the next question. 

• You may change an answer at any time by using the arrow keys to move 
the underline under the response you do want, and then hitting <enter>. 

6. Type your response to the open-ended questions (question s 7 & 8). 

7. Once you have completed a survey, press the “mail” icon |^3I • This will 
submit your ratings, and highlight the next team member in the team list. 

8. Repeat this process until you have completed rating all your team members. 

9. Once you have completely finished your ratings, please press the 
“walk away” icon 


10. Please answer the two feedback questions to help us improve this process. 





Helpful Hints 


• If you see this with a persons name 

* = team leader 

name in green = you have already rated this person 

• If you see this in the left margin 

★ = you have already rated this person 

-/ = you pressed the space bar — ignore this. 

• You can search for a name by typing a last name into the search box 
found in the upper left comer of the screen, and pressing the “search” 
button. 

• If you highlight a name shown in the search, make sure it is 

the correct name/team combination you want to rate and then 
press the “compose” icon. 

• To return to the team list, you must press the “clear” icon 
(found on the same line as the search box). 


• You may collapse list of names or list of teams by pressing the 
“collapse” icon! 


APPENDIX A-2 
Opening Screen 



Lotus Notes * Peer Evaluation - 2. List ot Teams 
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APPENDIX A-3a 
Peer Evaluation Form 



PEER EVALUATION FORM 


Team Name: Team A 


Ratee Name: LastName, First name 

Department: DepartmentName 


1 • First name LastName finds common groun d with others and gains cooperation easily 


2 . 


3. 


LJ Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off (average) 
0 Almost there; does this well most of the time 
® Don't change a thing; excellent 
0 N/A or not enough information to evaluate 


First name LastName responds to my requ ests in a timely manner 


Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off (average) 
0 Almost there; does this well most of the time 
® Don't change a thing; excellent 
0 N/A or not enough information to evaluate 


First name LastName clearly articulates wh at s/he needs from me 


0 Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off (average) 
0 Almost there; does this well most of the time 
® Don't change a thing; excellent 
0 N/A or not enough information to evaluate 


4. First name LastName is an attentive and active listener. 


U Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off (average) 
® Almost there; does this well most of the time 
0 Don't change a thing; excellent 
0 N/A or not enough information to evaluate 


First name LastName zeros in on the critic al tasks required to get the job done. 

O Needs to focus improvement efforts here 
® Sometimes on track & sometimes off (average) 

0 Almost there; does this well most of4he time 
0 Don't change a thing; excellent 
0 N/A or not enough information to evaluate 


First name LastName does quality work (e.g. accurate, done right the first time, etc.) 


0 Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off (average) 
0 Almost there; does this well most of the time 
® Don't change a thing; excellent 
0 N/A or not enough information to evaluate 


Please make your responses to the following two questions as constructive and 
helpful to First name LastName as possible. 

7. What does First name LastName do well? 

Comment 2 for Question #7 

8. What could First name LastName do better? 

Comment 2 for Question #8 




APPENDIX A-3b 
Peer Evaluation Results 



PEER EVALUATION RESULTS for LastName, First name 


Team Name: Team A Department: DepartmentName 
Total number of responses: 2 

1 = Needs to focus improvement efforts here 

2 = Sometimes on track & sometimes off (average) 

3 = Almost there; does this well most of the time 

4 = Don't change a thing; excellent 

0 = N/A or not enough information to evaluate 

1 . First name LastName finds common ground with others and gains cooperation easily. 

Average response rates you as "Almost there; does this well most of the time/Don't change a 
thing; excellent." 

Number of response : 2 

Mean : 3.50 

Std. : 0.500 

Highest response : 4 

Lowest response : 3 

2. First name LastName responds to my requests in a timely manner. 

Average response rates you as "Almost there; does this well most of the time." 

Number of response : 2 
Mean : 3.00 

Std. : 1.000 

Highest response : 4 

Lowest response : 2 

3. First name LastName clearly articulates what s/he needs from me. 

Average response rates you as "Almost there; does this well most of the time." 

Number of response : 2 
Mean : 3.00 

Std. : 1 .000 

Highest response : 4 
Lowest response : 2 

4. First name LastName is an attentive and active listener. 

Average response rates you as "Almost there; does this well most of the time/Don't change a 
thing; excellent." 

Number of response : 2 
Mean : 3.50 

Std. : 0.500 

Highest response : 4 
Lowest response : 3 

5. First name LastName zeros in on the critical tasks required to get the job done. 

Average response rates you as "Almost there; does this well most of the time." 

Number of response : 2 
Mean : 3.00 

Std. : 1 .000 

. * Highest response- : 4 

Lowest response : 2 

6. First name LastName does quality work (e.g. accurate, done right the first time, etc.) 

Average response rates you as "Don't change a thing; excellent." 

Number of response : 2 
Mean : 4.00 

Std. : 0.000 


Highest response : 4 

Lowest response : 4 


7. What does First name LastName do well? 
Comment for Question #7 

Comment 2 for Question Ml 

8. What could First name LastName do better? 
Comment for Question U 8 

Comment 2 for Question #8 



APPENDIX A-4a 
Session Evaluation Form 



SESSION EVALUATION FORM 



1 . Do you th ink this was a good way to collect team ratings? 

0 Yes 

0 No 


2. Do you have any suggestions to improve this rating process in the future? 


Press to Quit 




APPENDIX A-4b 
Session Evaluation Feedback 



SESSION EVALUATION FEEDBACK (SUMMER ’95) 



Number of Team Members:294 
Number of Responses: 294 100.00% 


1 . Do you think this was a good way to collect team ratings? 


Yes: 

205 

69.73% 

No: 

76 

25.85% 

Abstain: 

13 

4.42% 


2. Do you have any suggestions to improve this rating process in the future? 

I said no to the question above because your not collecting "team ratings" your collecting info 
on individuals/not the team. If you want to know about team performance, the question should 
be "how is this team functioning/meeting their goals" 

, etc. 

No, I don’t think this is a fair and objective way to rate people. I think this rating system 
should be discontinued 

A better machine with a real mouse, or have it on a server so you can use your own machine, 
with which you should be more familiar. 

Ask MORE questions. I didn't think they were all the ones we needed to ask. 

Don't do it. Use the resources to train people in program management skills. 

Don't do it. I don't see the value in this approach to employee evaluation. 

Eliminate it entirely. This process destroys what leadership and empowerment we have. 
Eliminate peer ratings. Individual peers can never tell if a particular person doesn't respond to 
their needs due to lack of ability (or motivation) or if there are other priorities set by superiors. 

It 

is destructive to the organization to give people false positive or negative feedback due to the 
probability that most peers cannot know the full extentof a particular person's workload and 
priorities. 

Expand the number of responses. I didn't find the questions very defining for m ; ost people. 
Figure out a way to keep people from evaluating themselves (possibly multiple times) maybe 
with an anomymous code good one time only. Somehow take the vengful potential out of the 
ratings (two people up for same promotion, etc.) 

I AM CONCERNED THAT RATING PEERS COULD BE DESTRUCTIVE TO TEAM 
RELATIONSHIPS. 

I didn’t have the time to do a complete job-spent 50 minutes, needed more. 

I don’t know what to do about it, but I do not think it is fair to rate or have people rating you 
that you may be competing with for promotions. Unfortunately, in the real world some people 
are not always as honest as they should be. 

I felt if I answered questions 7 & 8 and the employee saw these answers they would 
automatically know who had made these statements, so I was reluctant to comment 
I interface very little with the NASA personnel on the team. It would be useful to rate the Prime 
personnel as well, 
i suppose tha 

I think the only thing this is going to accomplish is to hurt a bunch of peoples feelings. If this 
system was suppose to help evaluate a team the questions should have been directed to the 
team as a whole. Such as "how to you feel the VAIT functions as a whole." Each team lead 
should know where his/her personnel is lacking. If not he/she is a poor manager. The way it 
was set up might be helpful in doing a persons performance appraisal. 

I would be strongly opposed to using this rating in performance evaluations. The 

results are very subjective to individual personalities. The uncontrolled nature of the selection 


process is subject to minimal inputs for some individuals. A better review process are the team 
evaluation forms (150 questions) that NASA uses in their project management 
training classes. This evaluation is not subject to individual personality differences. 

It focuses on evaluating the team, not the individual. I would still be opposed to using 
this form in end of the year evaluations, however, since that could bias the results. 

I would prefer the immediate supervisor or "lead" who assigned your work responsibilities and 
works with you on a daily basis should be the individuals who should rate. 

If unable to scrap this form of evaluation, focus on team and individual products. 

Instead of doing the evaluation randomly you should have have to evaluate a certain group of 
people and then if the evaluator wants to do others then he or she should be able to do that. It 
also would be good for the Customer to evaluate the person providing the service, ie. since the 
systems AITs provide a lot of support to the Launch Packages and other system AITs it would 
be good if they evaluated the person that provides support to them. It would also be useful if 
the functional managers above the person being evaluated also were made to be a mandatory 
evaluator. 

As for the room conditions I would like to see some privacy walls put up so I don't have to 
worry if the person next to me is reading what I typed in. 

It would be helpful to understand the entire process 

how are the evaluation used, how are the evaluation factored into the overall performance 
evaluation process, does the person evaluated see each rating input, is it truly anonymous, etc. 
It would be helpful to understand the entire process 

how are these evaluations used, how are the evaluations factored into the performance 
evaluation process, does the person evaluated see each rating input, is it truly anonymous, etc. 
more questions related to "people skills", this being the single most imporatant factor in my 
opinion in the effectiveness of our job 

My concern is that each person on the team does not know/care/understand anyone else's 
priorities. Therefore, it may appear that the person is not responding to requests in a timely 
manner and the person is rated accordingly. This information should be used only as a guide 
for the individual. This data should not be the driving force for the manager's evaluation. 

Need to address leveling of "stone throwing" and the opposite tendency to only evaluate team 
members positively. 

Need to see how these ratings are used , first ,then will decide. More time should be spent 
getting rid of the current rating system and going to a pass fail system. Pass/Fail system would 
be much more effective , especially in the program Office where more people are giving 1 50% 
effort than are allowed to receive the outstanding rating. 

No, I do not agree with this rating process. I could not effectively rate people I work with 
because I do not receive products from them. Also, I don't think that people are always fair and 
objective with this type of rating system. I believe that personality conflicts and personal issues 
may influence the ratings. For this reason, I do not feel that this is a fair system to use for 
performance appraisals. Therefore, I do not believe that these ratings should be given to 
Supervisors. 

No,this was a good first cut at a peer evaluation. I think it may help to expand the number of 
people who respond. Maybe make it mandatory for all personnel, i don't like mandatory but it 
tends to be the only way to ensure everyone gets involved and therefore the statistical 
relevance of the data is assured, 
no. 

one-on-one 

Provide cubicles for privacy. 

Provides more information to evaluate team members 

Questions too vague. I believe in more direct approach to dealing with good and bad 
co-workers. Describing problems should be a face to face process not with a computer in the 
middle. 

Questions too vague. I believe in the more direct approach to discussing issues or praise with 



coworkers. I don't believe in having a computer in the middle. 

Questions were asked of persons, not teams. If one wishes to rate teams, one should ask 
questions on team performance, not individual performance. 

Ratings should be completed by team lead or immediate supervisor only, 
scrap it. 

Send a real person to each of the teams to see how they are doing, maybe even a spy from 
management. 

Should be screened by supervisor or HR before giving out. I'm concerned about morale, when 
folks are working so hard. Better to have people sign their name to their evals. if they aren't 
willing to stand behind it, shouldn't collect it. 

Should have tested this on a test group first. Trying out a new process on 300 people is not a 
great idea, especially when those folks are already putting in 60-80 hours a week, 
supervisor/subordinates (and vice versa) should not be rated - this process allows such 
the true working teams are mainly composed of contractors and matrixed in people - not 
program office. To truely do a peer evaluation, the true Team memebers must be accounted 
for. This process does not adequately address the true working teams, 
the mouse stinks, and the names were not highlighted after I finished so I guess none of my 
work was sent.... 

The multiple choice section needs more questions and better options. 

The open section should have more sections. 

The system (computer) is not as user friendly (moose and cursor) as it should be. 

The questions seem somewhat leading, and a particular set of answers does not clearly reflect 
the type of person, quality of work, or type of team member. A battery of significantly more 
questions would be required to do this, and time required to evaluate team members with 
significantly more questions would be prohibitive. 

The questions seemed kind of innocuous and easy to rate most people highly. I would prefer to 
be rated on the goals that were mutually agreed to in my performance plan. A survey of peers 
that had working experience with me on these performance criteria would then be very useful. 

/i think this format is too generic to provide specific feedback for improvement. 

The use of peer evaluation on this program took almost all of us totally by surprise. This 
doesn’t strike me as being very team-oriented. Communication evidently broke down 
somewhere. We would have been better able to schedule our time to support this if we had 
known more than a few days in advance. 

While there were a few briefings on the peer evaluation before it was implemented, none of 
them were after normal working hours. Therefore few people could attend. In the future, I 
would recommend these informational briefings also be offered after hours as well. Things 
would have been a lot clearer and less traumatic if they had. 

The Vehicle Office had had a discussion about the peer evaluation process during our spring 
retreat. The overall discussion was that it was beneficial, however, it was VERY time 
consuming when done correctly. We therefore decided not to do it. Yet we are required to do 
it with the result that our productive time is impacted. Many of the respondents are evaluating 
only one individual in the interest of time. If we are serious about doing this correctly we need 
to be able to record who we have evaluated to remind us who we have evaluated and who we 
haven't. Otherwise you risk a mediocre response. 

And expand the after hours time slots for the room. It is nearly impossible to get in here during 
normal work hours. This is mandatory. 

It still is not clear who exactly is a peer. By the broadest interpretation that would be almost 
everyone in the Vehicle office. This needs to be scoped better or at least clearer. Some 
examples would help. You could spend hours upon hours in here if you evaluated everyone you 
communicate or work with on a daily basis. 

This is a good beginning, and should be done. However, there are more formal/controlled 
methods for rating. I think this could cause problems because of the very easy accessability. 
Time will tell. Hey, this is a lot better than what you had. 
this is stupid 



This method has the potential to turn into a popularity contest. Working well with other is not 
always the most impoartant factor. You may have to make unpopular statements to get the job 
done right. 

This process is personal not professional. You can rate people you don't like snd hurt them and 
rate people you do like to hurt them. You can even rate yourself to bolster your own rating. 
This is more of a popularity contest than a rating system. 

This report focused on the structure of my ganizational team, but not on my actual "Peers". It 
may be more useful to get feedback from those people who are really peers instead of those 
people who are in my organizational team. The launch package team members all work on 
different things with different groups, and it would be better to get input for thos people who 
on 

working together on a specific element or launch package (i.e. Node, launch package 2A). 

Too many variables in the evaluation. 

Work will become a popularity contest between members in order to get good ratings. 
Unfortunately in this environment, we're here to get a job done. There's too much work to 
answer individual requests for unscheduled work (due to poor planning on one side) and it may 
seem that a person isn’t performing well to a few individuals. 
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1 . 

1 • To start, prompt the user to enter his or her name, then show list of teams that the person 
is on. At first, I didn’t know which teams I was on, according to your data base. 

2. Do not produce prompt when a person scrolls too far through an evaluation form. Wait until 
the mail icon is pressed, then prompt if a question was inadvertently skipped. The 
scrolled-too-far prompt is VERY annoying! 

A clearer statement of how easy the process is and how feedback is going to be delivered to 
the people being evaluated would motivate more participation. Everyone is so busy 
sometimesthat they don’t do this. I always feel this is valuable, if only to vent frustration. 

A number of active IPTs were not covered, also need to include contractors 
Add a question about technical competence. 

Add questions which pertain to completing critical milestones on time and within budget. 

Add security to make sure that we have good quality control on the data, 
allow you to go back and review your responses even though you mailed them 



Although some people don't like to answer (or hear the answers to) open-ended qu 
estions, I think you get the most valuable feedback from those types of 
questions. 

An indication of how much time is spent interfacing/working with the individual rated. 

Allow users to access the ratings they have received thus far. 

Ability to return to a ratee and revise rating (after rating others). 

Ask more questions. If you do not reguarly write personel reviews it is hard to compose an 
anonmous narration commenting on someones performance, 
a. Provide print capability.. 

b. Provide team name and questions to each team prior to the evaluation period: this will 

allow personnel to work on evaluations ahead of time and during other activities that don't 
require continuous full attention. The evaluations can then be taken to the evaluation 

room 

and typed in quickly. 

Better define "peers" to be evaluated. 

Can't think of anything at this time. 

Change the timing of the rating system. Everyone seems to be extremely busy with many 
priorities at this time 

I’ve heard comments that team members just don't have the time to participate now. 

Clarify the role this evaluation will have in our overall appraisal. Randomize the team rosters, 
the team members with names near the beginning of the alphabet will get more evaluations, 
because we run out of time. 

Concerned that the process is not set up to evaluate members on cross pollinated teams (i.e, 
the VAIT has representation from Ops, Safety, Utilizations, Business, etc yet they were not 
highlighted on the VAIT team). It would be good to get feedback from members of other teams 
rather than searching thru the "other" category. Due to small numbers in some IPTs or AITs, 
by process of elimination a person could get a general feel of how he or she was evaluated by 
members of the team. There should be a minimum number of evaluators required if you're 
going to publish results 

Consolidate as many teams as possible. Provide evaluation handbook, which guides the 
evaluator through the process and gives examples of constructive comments. Provide the 
survey on line to all computors so that people can perform from their desks. 

Do it more often!!! 

Do not include contractors, I can not see them accepting this process -- government and boeing 
thinking are not 

along the same wavelength. Matrix input could be useful. 

Don't allow as much time to evaluate. Deadlines work 

Don’t make it blind rating 

own up to what you put into the system. 

Some people are members on paper of teams but are never included for meetings or activities. 
Have a way to show who is actually an active participant and who is just a member. 

Don’t provide info, to supervisor. 

Encourage people to participate - don't pressure them. 

Provide positive reinforcements to participate, not negative ones. 

"The floggings will continue until morale improves." 

It isn’t right to provide a sanctioned path for people to give unsubstantiated, anonymous, 
negative comments to someone’s supervisor. 

Examine other sets of questions fo ask. 

expand the basic questions to be more comprehensive 

Find a better way to make sure people are listed with ALL of the teams they regularly support. 
The general 'catch-all' ca 

Flesh out teams more. I had to go to "Other" to review individuals outside of my immediate 
team, even though I'm a member of said team. 

For Business Mgmt Office, revisit this process when the organization is finally stabilized. 



For future releases you should porvide more muliple choice questions. Perhaps each team could 
taylor these questions to itself. The value in a tool such as this is helping the individual identify 
the areas where he/she needs to improve. Having questions taylored to the team would go a 
long way to this end. 

get real mouses, this interfaces drove me crazy 

Given the multi-dimensions of team interactions (especially for VAIT) it would help to put some 
criteria together for reviewing interactions with other team— i.e. we establish in the TEP's what 
other teams we work with and should probably make that a part of thisevaluation. 

Good system. The thing that needs improving is the team roster process. If a team doesn't 
meet at least once a month, it probably isn't being managed in a team manner. Need to 
establish a birth control method for teams, (maybe some euthenasia, too) 

Have more focused questions . . . these seemed a bit on the nebulous side. Also expand this 
process to include our Boeing teammates. 

Have more questions and then allow space to expand after each question! 

Have more questions similar to those in the performance plan, perhaps even tailored to a given 
team or teams. 

how can we make this faster??? and still collect good stuff??? 

I'm glad to see you incorporated the comments of the focus groups 

I'm not sure. I guess I'm only afraid of the possibility that the folks who think I'm working hard 
and doing a great job may not rate me, whereas, I’m sure the folks who don’t like me will 
definitely 

evaluate me. I don’t know the answer to this. 

I do not know what happens to this data after it is compiled, but it woul 
I say "Yes" provided these ratings are not a component of anyone’s official perf 
ormance evaluation. Otherwise, I think a more controlled system of input should 
be implemented. I say this because the current system allows one to rate one's s 
elf highly, while torpedoing his/her adversaries. 

I suspect that there are probably other questions that could be asked of 
people, but I haven't given it any thought and so I have no suggestions at 
this time. 

I think it is more important to evaluate your customers than your teammembers. 

I think that there are a lot of "Sacred Cow" procedures and documents that are not serving their 
original intent. I think that a similar anonymous survey of the "process" would convey to 
management ways to improve the efficiency of the teams 

I think the multiple choice format is too limiting, and I would feel it would be more constructive 
to have more narrative answers to questions. Even though this would take more time, it would 
be more effective. 

I would like an opportunity to answer again after I see what is done with the information after it 
is compiled (Who gets to see what? How will it be distributed? etc.). It might help to add a 
section to comment on the performance of a whole team, as each team seems to have a group 
personality. This area should be separate from the evaluation of the team lead, to prevent 
evaluations of the team lead as a single team member from being mixed with an overall team 
evaluation. 

I would publish a list of the questions in advance so people could formulate the 
ir responses and be prepared. to just input the ratings. This I feel would take 
less computer time. 

Improve the Human Computer Interface. Let everyone know how easy it is so that we won't 
put it off for so long 

include contractors, they are some of our best people 
include folks outside the program 

Include the remote field center folks. Currently, this data base doesn't contain the NASA MSFC 
and KSC folks that support Utilization. It would also be helpful if they had this system and they 
could evaluate us. 

instead of question 7 & 8, include a comment section for each question 



It is very hard to be critical of my teammates because they are all very good engineers. Six 
questions is maybe too few to get a feel for their skills. An intro on how this survey will be 
used would also be helpful for those of us who missed any classes or meetings on the 
evaliation. 

These laptops are difficult to work with (whine) 
it only took a half hour, not bad at all. 

It seems unfortunate that this came up a such a busy 

It should not be set up so that supervisors are rating their employees, since there is another 
system for doing that, unless this is used in place of it. The "others" category is not necessary 
and is inviting character assassinations that are nonproductive. 

List more of the sub-IPT’s. I submitted my team but it was not on the system. 

Make sure everyone understands the purpose and the process 
make the questionaire more meaningful 

Make the system more user friendly by requiring (and instructing users) in only one key for 
selections and data entry. Disable unnecessary keys. Tell people to start by finding them- 
selves in their own organization and rating the appropriate people there, and then go to 
"OTHER" and find anyone else they may wish to evaluate. 

Make tool available via Mac's at each person's desk so that we don’t have to come to this 
room to fill out the forms. 

May want to incorporate team health questions. 

Maybe some work on the questions. 

More flexibility in rating options 
open-ended responses 

More questions and finer resolution answers. 

Need real mouse -- touch pad really is difficult to use. 

Ask this question again after we see the results 
too hard to evaluate the overall process yet. 

Need to improve the survey. The way the answers are structured, they are negative responses 
with the exception of one of them (the "excellent" rating). Also, there could be more multiple 
choice questions covering a larger range of performance. 

need to involve primary peers outside Progarm. Many folks 'work more with peers outside the 
program than within. 

need to look at a different set of questions that relate to teams, there are differences, go back 

to establishing objectives of team ratings and develop question that meet those objectives. 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

No 

No 

No 

No 

No 

none 



None at this time. 

None. 

None. 

not at present 
Not at this time 
Not at this time. 

Not at this time. 

Not at this tme 

No, none. This is an excellant survey. 

No, tThis is a good team evaluation process. 

No. 

No. 

No. 

No. ..My problem was lack of keyboard famaliarization with this particular comput 
er which cleared once I used it for a while 

People should be listed with all the teams they regularly support. The "all" categor 
Probably more questions that are not so generalized with spaces to elaborate. 

Probably need a few more items but this is a good start. 

Provide a way to indicate on what activities or teams' you are rating a person. 

Noone participants or is able to participate in all the teams they are assigned to the 
same extent, and therefore for some teams attributes would be incomplete. Those attributes 
are filled in by experiences with that person while involved in other teams or activities for which 
the evaluator is not an official member. 

Provide access at each persons computer instead of a single room 

Provide more insight into how this will be used - 1 like the concept but people need insight into 
how and why this is being done 

Provide more questions that are specific to the roles and responsibilities of each team. 

Put the evaluation format on the server to allow us to do it at our own workstations 
Put this on a server som where so we can do it from our desks, 
questions should be tailored to the job being performed by that person 
reduce mouse movements required to get through this 

add questions about specific member capabilities such as leadership capability a 
nd conflict management 

Seems like more control over who can evaluate who is needed to ensure the integrity of the 
feedback. Logon with a password and inhibits placed over groups you should not be rating. 
Some teams have varied personnel, and was not able to rate since I don't work directly with 
them, even though we are on the same team 

Team roster needs to be better defined, whether it's a functional team or a cross-functional 
team. 

Tell me what the questions are in advance so I can prepare my responses. 

Include Boeing counterparts of my team in the process. 

The evaluation questions are very limited, there should be more questions to really provide a fair 
and accurate evaluation. 

The layout of these laptops is poor. Hands keep hitting the mouse buttons while typing. 

The rating process is very subjective and not detailed enough. Ratings of say 1 to 10 after each 
response to the questions has been selected may be even more insite on the ratings for the 
persons. More questions may also be more appropiate. 

- There must be some accountability for commets to prevent personal issues coming through in 
job evaluations. 

This method, providing computers in a designated room, is good. The evaluations, however, 
are entirely subjective and full of individual bias. This may not matter if the evaluations are 
confined to informal communication to help employees identify strengths and weaknesses 
through the eyes of their peers. 

This should be a true peer evaluation not a team member evaluation. 


If we want to evaluate by teams we should also evaluate by peers. This should be a true 
peer evaluation not a team member evaluation. 

If we want to evaluate by teams we should also evaluate by peers. 

this will depend upon the report we all get back and the usefulness of it. The questions 

seemed to be appropriate. 

To many different groups. Suggest you pick a level at which evaluations should be done and 
have a person in only one group. If we want to evaluate others outside this group then we can 
go to your "other" listing. 

Use Mac's in the future. 

We need to use our actual working teams. They are much more meaningful then the teams by 
organization. The entire concept of IPTs are that they are integrated across all of the program 
areas. That would provide more realistic feedback from our real co-workers. 

Yes, collect a list of question other than ones listed. Could not answer correctly for each 
person 

rated because questions did not pertain. 
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Ask this question after we see if the results are useful. 

I'm not sure that this is a good way to collect team ratings. I AM sure that it should not be tied 
to performance ratings. I also believe very strongly that it should never become part of a 
person's file in personnel. I'm concerned about what uses information like this could be put to, 
once it is all in a computer and can be tabulated and compiled - and preserved forever. 
Misunderstandings and even unfounded judgements by people with an ax to grind could come 
back to haunt someone for a long time. 

So as far as I'm concerned, "the proof of the pudding is in the eating", that is. I'm reserving 
judgement on benefits for the team until I see the results. But I'm still leery of having this kind 
of information and judgement calls in a computer. I'm concerned that it could be used at some 
future time by people with their own agendas - agendas not related to the team improvement 
purpose for which this was designed. 

I think that, for everyone's protection, the information gathered here should be given to the 
team member and his or her supervisor, and then wiped out of the record. 

Inform decision makers that culture change takes time and must be encouraged not directed in a 
"voluntary" mandatory way. 

Don't provide anonymous, negative comments to people's leads or supervisors. 



Make raters more accountable by requiring them to enter their names (even if they are 

kept confidential). This will ensure that inputs are not frivolous. Also, require that all team 

members rate others. Often, only negative feedback is reported unless specifically 

requested. 

no 

no 

not at this time 

switching from mouse to keys is confusing... is there a way to make it all one way or all the 
other? 

the team lists need a lot of work. Eg., a supervisor should not be evaluating the people who 
work for him/her as peers. 

this method has great potential, however the categories need to be expanded or a method for 
adding qualifiers provided 
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Opening Screen for Individual Evaluation 


Lotus Notes - Peer Evaluation - 1 996 - 2. List of teams\b. by Member 
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1. Department A 

1 .1 - LastName2. FirstName2 
1.2- LastName. FirstName 


2 OA - Program Office 


3. OB -Vehicle 

4. OC&OD - Operations and Utilization 

5. OE - Safety and Mission Assurance 

6. OF - Program Risk Management 

7. OG - Business Management 

8. OH - Phase 1 

9. 01 - International Partners 








APPENDIX B-2a 
Individual Evaluation Form 



INDIVIDUAL EVALUATION FORM 


Ratee Name: LastName2, FirstName2 Department: Department A 


COMMUNICATION 


1 . FirstName2 LastName2 clearly articulates what s/he needs from me. 

(is easily understood, gets messages across effectively, both orally and in writing) 


O" Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
9 Don't change a thing 
0 N/A or not enough information to evaluate 


2. FirstName2 LastName2 pays full attention to me and the messages I convey. 
(actively listens, checks for understan ding, this includes written requests) 


"O' Needs to focus improvement efforts here 
0 Sometimes on track fit sometimes off 
0 Almost there; does this well most of the time 
® Don't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about FirstName2 LastName2 regarding communication? 

RELIABILITY 


3. FirstName2 LastName2 responds to my requests in a timely manner. 
(is responsive, is accessible) 

O Needs to focus improvement efforts here 
0 Sometimes on track fit sometimes off 
0 Almost there; does this well most of the time | 

® Don’t change a thing 
0 N/A or not enough information to evaluate 


4. FirstName2 LastName2 does quality work, 
(is accurate, is done right the first time) 


(U Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Don't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about FirstName2 LastName2 regarding reliability? 






LEADERSHIP/MEMBERSHIP 


5. FirstName2 LastName2 takes initiative with his/her work. 

(sees what needs to be done, is decisive, can lead others when necessary, 
can comfortably handle risk and unce rtainty) 

O Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
0 Don’t change a thing 
0 N/A or not enough information to evaluate 


6. FirstName2 LastName2 finds common ground with others and gains cooperation easily. 
(is a team supporter, knows when to get help, handles conflict effectively) 

Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
0 Don't change a thing 
0 N/A or not enough information to evaluate 


7. FirstName2 LastName2 zeros in on the critical tasks required to get the job done. 
(is organized, doesn't waste time) 

”0 Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
0 Don't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about FirstName2 LastName2 regarding leadership/membership? 
DEVELOPMENT 


8 . 


FirstName2 LastName2 continuously improves him/her-self. 

(seeks feedback, takes constructive criticism well, develops improvement plans for self, 
watches others for their reactions, ta kes training) 

0 Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
0 Don't change a thing 
0 N/A or not enough information to evaluate 


9 . 


FirstName2 LastName2 seeks to develop his/her team. 

(gives constructive criticism, shares experiences with team, supports the team effort) 






0 Needs to focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Don't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about FirstName2 LastName2 regarding development? 


I 

t 


OPEN ENDED 


10. What else does FirstName2 LastName2 do well? 
Comment for Question #10 

1 1 . What else could FirstName2 LastName2 do better? 
Comment for Question #1 1 




APPENDIX B-2b 
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FEEDBACK RESULTS for LastName2, FirstName2 

Department: Department A Total number of responses:2 


1 = Focus improvement efforts here 

2 = Sometimes on track & sometimes off 

3 = Almost there; does this well most of the time 

4 = Don't change a thing 

0 = N/A or not enough information to evaluate 

COMMUNICATION 


FirstName2 LastName2 clearly articulates what s/he needs from me. 

(is easily understood, gets messages across effectively, both verbally and in writing) 

Average response rates you as "Almost there; does this well most of the time/Don't change a thing. 

HI. I n 


Number of response 
Mean 
Std. 

Highest response 
Lowest response 


2 

3.50 

0.500 

4 

3 


2 . 


FirstName2 LastName2 pays full attention to me and the messages I convey. 

(actively listens, checks for understanding, this includes written requests) 

Average response rates you as "Almost there; does this well most of the time/Don't change a 
thing." 


Number of response 

Mean 

Std. 

Highest response 
Lowest response 


2 

3.50 

0.500 

4 

3 


RELIABILITY 


3. 


FirstName2 LastName2 responds to my requests in a timely manner. 

(is responsive, is accessible) 

Average response rates you as "Almost there; does this well most of the time/Don't change a 
thing." 


Number of response 

Mean 

Std. 

Highest response 
Lowest response 


2 

3.50 

0.500 

4 

3 


4. FirstName2 LastName2 does quality work. 

(is accurate, is done right the first time) 

Average response rates you as "Almost there; does this well most of the time." 


Number of response 
Mean 
Std. 

Highest response 
Lowest response 


2 

3.00 

1.000 
4 

2 


LEADERSHIP/MEMBERSHIP 


5. FirstName2 LastName2 takes initiative with his/her work. 

(sees what needs to be done, is decisive, can lead others when necessary, 
can comfortably handle risk and uncertainty) 

Average response rates you as "Almost there; does this well most of the time." 


Number of response 
Mean 
Std. 

Highest response 
Lowest response 


2 
3.00 
0.000 
3 
3 


6 . 


FirstName2 LastName2 finds common ground with others and gains cooperation easily. 

(is a team supporter, knows when to get help, handles conflict effectively) 

Average response rates you as "Almost there; does this well most of the time/Don't change a 
thing." 


Number of response 

Mean 

Std. 

Highest response 
Lowest response 


2 

3.50 

0.500 

4 

3 


7. 


FirstName2 LastName2 zeros in on the critical tasks required to get the job done. 

(is organized, doesn’t waste time) 

Average response rates you as "Almost there; does this well most of the time/Don’t change a 
thing." 


Number of response 

Mean 

Std. 

Highest response 
Lowest response 


2 

3.50 

0.500 

4 

3 


DEVELOPMENT 


8 . 


FirstName2 LastName2 continuously improves him/her-self. 

(seeks feedback, takes constructive criticism well, develops improvement plans for self, 
watches others for their reactions, takes training) 

Average response rates you as "Almost there; does this well most of the time/Don't change a 
thing." 


Number of response 

Mean 

Std. 

Highest response 
Lowest response 


2 

3.50 

0.500 

4 

3 


9. FirstName2 LastName2 seeks to develop his/her team. 

(gives constructive criticism, shares experiences with team, supports the team effort) 
Average response rates you as "Don't change a thing." 

Number of response : 2 
Mean : 4.00 

Std. : 0.000 

Highest response : 4 



Lowest response : 4 


OPEN ENDED COMMENTS 

Comments about FirstName2 LastName2 regarding communication? 

Comments about FirstName2 LastName2 regarding reliability? 

Comments about FirstName2 LastName2 regarding leadership/membership? 

Comments about FirstName2 LastName2 regarding development? 

OPEN ENDED 

1 0. What else does FirstName2 LastName2 do well? 

Comment #2 for Question #1 0 
Comment for Question #10 


1 1 . What else could FirstName2 LastName2 do better? 
Comment #2 for Question #11 
Comment for Question #11 



APPENDIX B-3 

Opening Screen for Team Evaluations 



Lotus Notes - Peer Evaluation - 1 996 - 2. List of teams\a. by Team 


EHe Edit ¥iew Mail Compose Jcxt Tflols Design Window help 



1. Department A 

1.1 - Team A (NASA Team Leader/Other Team Leader) 

2. OA - Program Office 


13. OB- Vehicle 


4. OC&OD • Operations and Utilization 

5. OE - Safety & Mission Assurance 

6. OF- Program Risk Management 

7. OG - Business Management 

8. OH - Phase 1 

9. 01 - International Partners 











APPENDIX B-4a 
Member Evaluation Form 



Team Name: Team A 
COMMUNICATION 


MEMBER EVALUATION FORM 

Department: Department A 


1 . Within the Team A we communicate effectively. 

(we share needed information, get m essages across effectively; both orally and in writing) 

O Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn't change a thing 
0 N/A or not enough information to evaluate 


2. Within the Team A we pay attention to each other. 

(we actively listen, check for underst anding; with both oral & written requests) 


u Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding communication? 
ROLES AND RESPONSIBILITIES 


3. Within the Team A we know who is responsible for each task. 

(we have a clear charter, clearly defin ed products, clearly defined roles and responsibilities) 
I L/ Focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 


4. Within the Team A our leader is effective. 

(can lead others, is decisive, plans pr ojects skillfully, can get the best out of team members) 


"O Focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding roles and responsibility? 


CONFLICT MANAGEMENT 




5. Within the Team A we cooperate and resolve conflicts constructively. 

(we know when it's best to comprom ise, collaborate in work being done) 

w Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time | 

® Shouldn’t change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding conflict management? 


DECISION MAKING 


6. Within the Team A we have a useful decision making process. 
(it is effective, organized, timely, acc eptable) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn’t change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding decision making? 
TRUST/RESPECT 


7. Within the team Team A we treat each other with fairness, trust, and respect. 

show common courtesy to each other) 


What comments would you like to make about Team A regarding trust/respect? 
TEAMING ENVIRONMENT 

8. Within the Team A we are committed to the team effort. 

(we get the job done, support each o ther) 

O Focus improvement efforts here 
0 Sometimes on track & sometimes off 

Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 



(we create a supportive environment, 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn't change a thing 
0 N/A or not enough information to evaluate 


9. Within the Team A we recognize each others* accomplishments. 

(we are appreciative of work well done, celebrate both individual and team 





accomplishments, reward milestones ) 

O Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
0 Shouldn't change a thing 
® N/A or not enough information to evaluate 


10. The Team A is accom plishing what the program needs it to accomplish. 

T3 Strongly Disagree 

0 Disagree 
0 Moderately Disagree 
0 Moderate 
0 Moderately Agree 
® Agree 
0 Strongly Agree 

1 1 . would rate my overal satisfaction with the Team A as: 

0 Completely Unsatisfied 

0 Unsatisfied 

0 Moderately Unsatisfied 
0 Moderate 
® Moderately Satisfied 
0 Satisfied 
0 Completely Satisfied 

What comments would you like to make about Team A regarding teaming environment? 


OPEN ENDED 

1 2. What else does the Team A do well? 
Comment #1 for Question #12 

13. What else could the Team A do better? 
Comment #1 for Question #13 





APPENDIX B-4b 
Team Member Feedback Results 



TEAM MEMBER FEEDBACK RESULTS 

for : Department A 
Total number of responses: 2 


1 = Focus improvement efforts here 

2 = Sometimes on track & sometimes off 

3 = Almost there; does this well most of the time 

4 = Shouldn't change a thing 

0 = N/A or not enough information to evaluate 

COMMUNICATION 


1. Within the [TEAM NAME] we communicate effectively. 

(we share needed information, get messages across effectively; both orally and in writing) 
Average response rates you as "Shouldn’t change a thing." 

Number of response : 2 

Mean : 4.00 

Std. : 0.000 

Highest response : 4 

Lowest response : 4 


2. Within the [TEAM NAME] we pay attention to each other. 

(we actively listen, check for understanding; with both oral & written requests) 
Average response rates you as "Shouldn't change a thing." 


Number of response 

Mean 

Std. 

Highest response 
Lowest response 


2 

4.00 

0.000 

4 

4 


ROLES AND RESPONSIBILITIES 


Within the [TEAM NAME] we know who is responsible for each task. 

(we have a clear charter, clearly defined products, clearly defined roles and responsibilities) 
Average response rates you as "Almost there; does this well most of the time." 


Number of response 

Mean 

Std. 

Highest response 
Lowest response 


2 

3.00 

0.000 

3 

3 


4. Within the [TEAM NAME] our leader is effective. 

(can lead others, is decisive, plans projects skillfully, can get the best out of team members) 
Average response rates you as "Almost there; does this well most of the time." 


Number of response 
Mean J 

Std. 

Highest response 
Lowest response 


2 
3.00 
0.000 
3 
3 


CONFLICT MANAGEMENT 


5. Within the [TEAM NAME] we cooperate and resolve conflicts constructively, 
(we know when it’s best to compromise, collaborate in work being done) 
Average response rates you as "Shouldn't change a thing." 


Number of response 
Mean 
Std. 

Highest response 
Lowest response 


2 

4.00 

0.000 

4 

4 


DECISION MAKING 


6. Within the [TEAM NAME] we have a useful decision making process, 
(it is effective, organized, timely, acceptable) 

Average response rates you as "Shouldn't change a thing." 


Number of response 
Mean 
Std. 

Highest response 
Lowest response 


2 

4.00 

0.000 

4 

4 


TRUST/RESPECT 


7. Within the [TEAM NAME] we treat each other with fairness, trust, and respect, 
(we create a supportive environment, show common courtesy to each other) 
Average response rates you as "Shouldn't change a thing." 


Number of response 
Mean 
Std. 

Highest response 
Lowest response 


2 

4.00 

0.000 

4 

4 


TEAMING ENVIRONMENT 


8 . 


Within the [TEAM NAME] we are committed to the team effort. 

(we get the job done, support each other) 

Average response rates you as "Sometimes on track & sometimes off /Almost there; does this 
well most of the time." 


Number of response 

Mean 

Std. 

Highest response 
Lowest response 


2 

2.50 

0.500 

3 


2 


9. Within the [TEAM NAME] we recognize each others' accomplishments. 

(we are appreciative of work well done, celebrate both individual and team 
accomplishments, reward milestones) 

Average response rates you as "Sometimes on track & sometimes off." 
Number of response : 1 


Mean : 2.00 

Std. : 0.000 

Highest response : 2 

Lowest response : 2 


10. The [TEAM NAME] is accomplishing what the program needs it to accomplish. 

Count Percentage 

1 = Strongly Disagree 0 0.00% 

2 = Disagree 0 0.00% 

3 = Moderately Disagree 0 0.00% 

4= Moderate 0 0.00% 

5 = Moderately Agree 0 0.00% 

6= Agree 2 100.00% 

7 = Strongly Agree 0 0.00% 


11.1 would rate my overall satisfaction with the [TEAM NAME] as: 


Count 

1 = Completely Unsatisfied 0 

2 = Unsatisfied 0 

3 = Moderately Unsatisfied 0 

4 = Moderate 1 

5 = Moderately Satisfied 1 

6 = Satisfied 

7 = Completely Satisfied 0 



0 . 00 % 

50.00% 

50.00% 

0 0 . 00 % 
0 . 00 % 



OPEN ENDED COMMENTS 


Comments about Team A regarding communication? 

Comments about Team A regarding roles and responsibility? 

Comments about Team A regarding conflict management? 

Comments about Team A regarding decision making? 

What comments would you like to make about Team A regarding trust/respect? 

What comments would you like to make about Team A regarding teaming environment? 


OPEN ENDED 


1 2. What else does the Team A do well? 
Comment #2 for Question #12 
Comment #1 for Question #12 

1 3. What else could the Team A do better? 
Comment #2 for Question #13 
Comment #1 for Question #13 



APPENDIX B-5a 
Customer Evaluation Form 



Team Name: Team A 


CUSTOMER EVALUATION FORM 

Department: Department A 


COMMUNICATION 


1 . The Team A communicates effectively with me. 

(they .share needed information, get m essages across effectively; both orally and in writing) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
9 Shouldn't change a thing 
0 N/A or not enough information to evaluate 

2. The Team A pays attention to me. 

(they actively listen, check for unders tanding; with both oral & written requests) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
0 Shouldn't change a thing 
® N/A or not enough information to evaluate 



3. I understand the roles and responsibilities of the Team A. 
(charter, products, roles and responsibilities) 


Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding communication? 
CONFLICT MANAGEMENT 


4. The Team A cooperates and resolves conflicts constructively. 

(they know when it's best to compro mise, collaborate in work being done) 

O Focus improvement efforts here 
® Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding conflict management? 


DECISION MAKING 







5. The Team A makes good decisions. 

(stable, based on facts and reason, ti mely) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time | 

0 Shouldn’t change a thing 
0 N/A or not enough information to evaluate 

What comments would you like to make about Team A regarding decision making? 

TRUST/RESPECT 

6. The Team A treats others with fairness, trust, and respect. 

(they create a supportive environmen t, show common courtesy) 


O' Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding trust/respect? 
TEAMING ENVIRONMENT 


7. The Team A gets the job done effectively. 

(they know what needs to be done a nd do it, step up to challenges when necessary) 


0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn’t change a thing 
0 N/A or not enough information to evaluate 


Members of the Team A are accessible when needed. 
(they are easy to reach, responsive, a ttentive) 


"O' Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn't change a thing 
0 N/A or not enough information to evaluate 


"What comments would you like to make about Team A regarding teaming environment? 


OPEN ENDED 


9. What else does the Team A do well? 
Comment #1 for Question #9 







10. What else could the Team A do better? 
Comment #1 for Question #10 



APPENDIX B-5b 
Customer Feedback Results 



CUSTOMER FEEDBACK RESULTS 

for: Team A 

Department: Department A 
Total number of responses: 1 



1 = Focus improvement efforts here 

2 = Sometimes on track & sometimes off 

3 = Almost there; does this well most of the time 

4 = Shouldn't change a thing 

0 = N/A or not enough information to evaluate 

COMMUNICATION 


1. 

The Team A communicates effectively with me. 


(they share needed information, get messages across effectively; both orally and in writing) 
Average response rates you as "Shouldn't change a thing." 


Number of response 

1 


Mean 

4.00 


Std. 

0.000 


Highest response 

4 


Lowest response 

4 

2. 

The Team A pays attention to me. 



(they actively listen, check for understanding; with both oral & written requests) 
Average response rates you as "N/A or not enough information to evaluate." 


Number of response 

0 


Mean 

0.00 


Std. 

Highest response 
Lowest response 

0.000 

3. 

i 

1 understand the roles and responsibilities of the Team A. 

(charter, products, roles and responsibilities) 

Average response rates you as "Shouldn't change a thing." 


Number of response 

1 


Mean 

4.00 


Std. 

0.000 


Highest response 

4 

i 

Lowest response 

4 

CONFLICT MANAGEMENT 



4. The Team A cooperates and resolves conflicts constructively. 

(they know when it's best to compromise, collaborate in work being done) 
Average response rates you as "Sometimes on track & sometimes off." 
Number of response : 1 


Mean : 2.00 

Std. : 0.000 

Highest response : 2 

Lowest response : 2 


i 


DECISION MAKING 




5. The Team A makes good decisions. 


(stable, based on facts and reason, timely) 

Average response rates you as "Almost there; does this well most of the time." 

Number of response 

1 

Mean 

3.00 

Std. 

0.000 

Highest response 

3 

Lowest response 

3 

TRUST/RESPECT 



6. The Team A treats others with fairness, trust, and respect. 

(they create a supportive environment, show common courtesy) 
Average response rates you as "Shouldn't change a thing." 


Number of response 

i 

Mean 

4.00 

Std. 

0.000 

Highest response 

4 

Lowest response 

4 


TEAMING ENVIRONMENT 

7. The Team A gets the job done effectively. 

(they know what needs to be done and do it, step up to challenges when necessary) 
Average response rates you as "Shouldn't change a thing." 

Number of response : 1 


Mean : 4.00 

Std. : 0.000 

Highest response : 4 

Lowest response : 4 


8. Members of the Team A are accessible when needed. 

(they are easy to reach, responsive, attentive) 

Average response rates you as "Shouldn't change a thing." 
Number of response : 1 


Mean : 4.00 

Std. : 0.000 

Highest response : 4 

Lowest response : 4 


OPEN ENDED COMMENTS 


Comments about Team A regarding communication? 
Comments about Team A regarding conflict management? 
Comments about Team A regarding decision making? 
Comments about Team A regarding trust/respect? 
Comments about Team A regarding teaming environment? 


OPEN ENDED 


9. What else does the Team A do well? 
Comment #1 for Question #9 

10. What else could the Team A do better? 
Comment #1 for Question #10 


Which of your customers evaluated you? 



APPENDIX B-6a 
Supplier Evaluation Form 


Team Name: Team A 


SUPPLIER EVALUATION FORM 

Department: Department A 


COMMUNICATION 


1 . The Team A communicates effectively with me. 

(we share needed information, get m essages across effectively; both orally and in writing) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn't change a thing 
0 N/A or not enough information to evaluate 

2. The Team A pays attention to me. 

(they actively listen, check for unders tanding; with both oral & written requests) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 

3. I understand the roles and responsibilities of the Team A. 

(charter, products, roles and responsibilities) 


Focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 

What comments would you like to make about Team A regarding communication? 
CONFLICT MANAGEMENT 


4. The Team A cooperates and resolves conflicts constructively. 

(they know when it's best to compro mise, collaborate in work being done) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn't change a thing 
~ 0 N/A or not enough information to evaluate 

What comments would you like to make about Team A regarding conflict management? 


DECISION MAKING 






5. The Team A makes good decisions. 

(stable, based on facts and reason, ti mely) 

0 Focus Improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time | 

0 Shouldn't change a thing 
® N/A or not enough information to evaluate 

What comments would you like to make about Team A regarding decision making? 


TRUST/RESPECT 

i — — 

i 

6. The Team A treats others with fairness, trust, and respect. 

(they create a supportive environmen t, show common courtesy) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding trust/respect? 


TEAMING ENVIRONMENT 


7. The Team A gets the job done effectively. 

(they are easy to reach, responsive, a ttentive) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
O Shouldn't change a thing 

0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding teaming environment? 
OPEN ENDED 


9. What else does the Team A do well? 
Comment #1 for Question #9 

10. What else could the Team A do better? 
Comment #1 for Question #10 





APPENDIX B-6b 
Supplier Feedback Results 




DECISION MAKING 


5. The Team A makes good decisions. 


(stable, based on facts and reason, timely) 

Average response rates you as "N/A or not enough information to evaluate." 

Number of response 

0 

Mean 

0.00 

Std. 

Highest response 
Lowest response 

0.000 

TRUST/RESPECT 




6. The Team A treats others with fairness, trust, and respect. 

(they create a supportive environment, show common courtesy) 

Average response rates you as "Almost there; does this well most of the time." 

Number of response 

1 

Mean 

3.00 

Std. 

0.000 

Highest response 

3 

Lowest response 

3 

TEAMING ENVIRONMENT 



7. The Team A gets the job done effectively. 

(they are easy to reach, responsive, attentive) 

Average response rates you as "Almost there; does this well most of the time." 
Number of response : 1 


Mean : 3.00 

Std. : 0.000 

Highest response : 3 

Lowest response : 3 


OPEN ENDED COMMENTS 


Comments about Team A regarding communication? 
Comments about Team A regarding conflict management? 
Comments about Team A regarding decision making? 
Comments about Team A regarding trust/respect? 
Comments about Team A regarding teaming environment? 


OPEN ENDED 

8. What else does the Team A do well? 
Comment #1 for Question #9 

9. What else could the Team A do better? 
Comment #1 for Question #10 

Which of your customers evaluated you? 


APPENDIX B-7a 
Other Evaluation Form 



Team Name: Team A 


OTHER EVALUATION FORM 

Department: Department A 


COMMUNICATION 


1 . The Team A communicates effectively with me. 

(they share needed information, get m essages across effectively; both orally and in writing) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn’t change a thing 
0 N/A or not enough information to evaluate 

2. The Team A pays attention to me. 

(they actively listen, check for unders tanding; with both oral & written requests) 

0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 

0 Shouldn’t change a thing 
0 N/A or not enough information to evaluate 

3. I understand the roles and responsibilities of the Team A. 

(charter, products, roles and respons ibilities) 

W Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 

What comments would you like to make about Team A regarding communication? 

CONFLICT MANAGEMENT 


4. The Team A cooperates and resolves conflicts constructively. 

(they know when it*s best to compro mise, collaborate in work being done) 

Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn't change a thing 
* 0 N/A or not enough information to evaluate 

What comments would you like to make about Team A regarding conflict management? 


DECISION MAKING 






5. The Team A makes good decisions. 

(stable, based on facts and reason, ti mely) 


u Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
® Shouldn't change a thing 
0 N/A or not enough information to evaluate 


What comments would you like to make about Team A regarding decision making? 
TRUST/RESPECT 


6. The Team A treats others with fairness, trust, and respect. 

(they create a supportive environmen t, show common courtesy) 


"0 Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 
0 Shouldn't change a thing 
® N/A or not enough information to evaluate 

l 

What comments would you like to make about Team A regarding trust/respect? 

TEAMING ENVIRONMENT 


7. The Team A gets the job done effectively. 

(they know what needs to be done a nd do it, step up to challenges when necessary) 


'O Focus improvement efforts here 
0 Sometimes on track & sometimes off 
0 Almost there; does this well most of the time 

® Shouldn't change a thing 

0 N/A or not enough information to evaluate 


8. Members of the Team A are accessible when needed. 

(they are easy to reach, responsive, a ttentive) 

O Focus improvement efforts here 
0 Sometimes on track & sometimes off 
® Almost there; does this well most of the time 
0 Shouldn't change a thing 
0 N/A or not enough information to evaluate 

What comments would you like to make about Team A regarding teaming environment? 


OPEN ENDED 


9. What else does the Team A do well? 
Comment #1 for Question #9 



10. What else could the Team A do better? 
Comment #1 for Question #10 



APPENDIX B-7b 
Other Feedback Results 



OTHER FEEDBACK RESULTS 

for: Team A 

Department: Department A 
Total number of responses: 1 


1 = Focus improvement efforts here 

2 = Sometimes on track & sometimes off 

3 « Almost there; does this well most of the time 

4 = Shouldn't change a thing 

0 = N/A or not enough information to evaluate 

COMMUNICATION 


1 . The Team A communicates effectively with me. 

(they share needed information, get messages across effectively; both orally and in writing) 
Average response rates you as "Shouldn't change a thing." 

Number of response : 1 


Mean : 4.00 

Std. : 0.000 

Highest response : 4 

Lowest response : 4 


2. The Team A pays attention to me. 

(they actively listen, check for understanding; with both oral & written requests) 
Average response rates you as "Almost there; does this well most of the time." 
Number of response : 1 


Mean : 3.00 

Std. : 0.000 

Highest response : 3 

Lowest response : 3 


3. I understand the roles and responsibilities of the Team A. 

(charter, products, roles and responsibilities) 

Average response rates you as "Focus improvement efforts here." 
Number of response : 1 


Mean : 1.00 

Std. : 0.000 

Highest response : 1 

Lowest response : 1 



CONFLICT MANAGEMENT 


4. The Team A cooperates and resolves conflicts constructively. 

(they know when it's best to compromise, collaborate in work being done) 
Average response rates you as "Shouldn't change a thing." 

Number of response : 1 


Mean : 4.00 

Std. : 0.000 

Highest response : 4 

Lowest response : 4 


DECISION MAKING 


5. The Team A makes good decisions. 

(stable, based on facts and reason, timely) 

Average response rates you as "Shouldn't change a thing." 
Number of response : 1 


Mean : 4.00 

Std. : 0.000 

Highest response : 4 

Lowest response : 4 


TRUST/RESPECT 


6. The Team A treats others with fairness, trust, and respect. 

(they create a supportive environment, show common courtesy) 

Average response rates you as "N/A or not enough information to evaluate." 
Number of response : 0 
Mean : 0.00 

Std. : 0.000 

Highest response : 

Lowest response : 

TEAMING ENVIRONMENT 


7. The Team A gets the job done effectively. 

(they know what needs to be done and do it, step up to challenges when necessary) 
Average response rates you as "Shouldn’t change a thing." 

Number of response : 1 


Mean : 4.00 

Std. : 0.000 

Highest response : 4 

Lowest response : 4 


.8. Members of the Team A are accessible when needed. 

(they are easy to reach, responsive, attentive) 

Average response rates you as "Almost there; does this well most of the time." 
Number of response : 1 

Mean : 3.00 

Std. : 0.000 

Highest response : 3 



Lowest response : 3 


OPEN ENDED COMMENTS 


Comments about Team A regarding communication? 

Comments about Team A regarding conflict management? 

Comments about Team A regarding decision making? 

Comments about Team A regarding trust/respect? 

Comments about Team A regarding teaming environment? 
OPEN ENDED 

9. What else does the Team A do well? 

Comment #1 for Question #9 

10. What else could the Team A do better? 

Comment #1 for Question #10 


Which of your customers evaluated you? 


APPENDIX B-8a 
Session Evaluation Form 



■ WE WANT TO KNOW! 

Please tell us what you think of the individual and team evaluation process. 
Answer the following questions if applicable. 

Individual Evaluation Process 


1 . Do you feel you get useful information about you rself from this feedback? 
0 1 . Strongly Disagree (Not at all useful) 

0 2. Disagree (Not Useful) 

0 3. Neutral (Some of this info, is useful) 

0 4. Agree (Useful) 

0 5. Strongly Agree (This info, is very useful) 


2. What could we do to improve the process of evaluating individuals? 
Team Evaluation Process 

3. Do you feel that teams will get useful information from this process? 
0 1 . Strongly Disagree (Not at all useful) 

0 2. Disagree (Not Useful) 

0 3. Neutral (Some of this info, is useful) 

0 4. Agree (Useful) 

0 5. Strongly Agree (This info, is very useful) 


4. What could we do to improve the process of evaluating teams? 


Press to Quit 






APPENDIX B-8b 

Session Evaluation Feedback (Individual Process) 



SESSION EVALUATION FEEDBACK (SPRING ’96) 

Number of Rators:412 

Number of Responses: 247 59.95% 


Individual Evaluation Process 


1 . Do you feel you get useful information about yourself from this feedback? 


1 . Strongly Disagree (Not at all useful) 

15 

6.73% 

2. Disagree (Not Useful) 

15 

6.73% 

3. Neutral (Some of this info, is useful) 

111 

49.78% 

4. Agree (Useful) 

60 

26.91% 

5. Strongly Agree (This info, is vey useful) 

22 

9.87% 


2. What could we do to improve the process of evaluating individuals? 

Get rid of it. 

I matrix into the team. In this evaluation process, all I can do is input. Why are we only half-part 
of the process when we’re fully part of the team? 

If I have a specific beef, I take it up the team management structure, (and I have). The 
fill-in-the-blank routine makes it hard to communicate a specific message or suggestion. Only 
the free-form entry really seems worth our while to enter. 

Not useful to me, I am a Contract engineer and am not included in this process aidividually. 
Suggets you add contractors to the individual appraisal process 

Since, with all of your assurances I still feel that my answers can be tracked down to me and I 
am not about to risk my job I won't be really accurate. 

there are no controls or filtering. This program creates such stress that you are likely to get 
folks who lash out because not only are they frustrated in general, but they also are forced to 
spend valuable time doing what is perceived to be a non-value added activity. I haven't heard 
one person say one thing positive about this process. Given that environment, the data is very 
suspect. 

An On-line form at my office PC 

et their supervisors rate them based on their productivity not on personal feelings which that 
individuals may have regarding a team leader. It is not possible to control someones personal 
feelings that may have been provocate to give a bad rating when you no nothing of the 
circumstances that may have caused a bad report. 

Limit the number of people each can and will evaluate. 

the best evaluation of an individual comes from his/her supervisor. Improve that process. 

There is not enough time to rate all the folks I work with. I have a feeling other evaluators have 
the same feeling and don't spend much time on my own evaluation. 

This type of reveiw can be dangerous. From comments, one can deduce who provided negative 
feedback. This can cause hardfeelings in the organization. I think focus groups where folks get 
together to work out problems are more productive. Team leads need to hold these type of 
"retreats with their team and take issues forward issues to management. Make this part of the 
leaders and supervisors performance plans. A standard template and report could be required. 
Add Boeing Prime individuals. 

Alter the format. 

do it only once per year so people can take adequate time to provide constructive criticism 
twice a year means that people may rush through knowing that they can do it again in 6 mos 
Do not allow supervisors to evaluate their employees. This distorts the feedback. 

Don't make us do this in the middle of crises period, you may get better response 

Due to limited time, couldn't evaluate everyone I'd like. Would be great if this could be accessed 

from the PCs. 

Expand the question base. Perhaps have more questions and allow evaluator to select which 
questions he wants to answer for a specific individual. 


Go one level lower so that the subteams can evaluate thems within a team. 

Have a list of adjectives describing people and their behavior that one could select applicable 
traits and thus provide a word-picture of an individual. For instance, bright, dull, cooperative, 
reluctant, tardy, punctual, brusque, cordial, 

Have the personnel develop a list of ALL individuals they have interacted with and make that list 
mandatory that they evaluate everyone on the list. By doing this the responses that we get as 
feedback should average out the extreme responses. 

Have the team members create a list of all personnel they interact with. It should then be 
mandatory to evaluate everyone on your list lie. at least everyone in your immediate office). This 
will average out the extreme responses and provide better feedback. 

I am not sure this is helpful for evaluating individuals because the results can be easily 
misleading if someone has an axe to grind against someone. 

I thning this is a good system for a cursory level evaluation of a large group of people. Not sure 
of the effectiveness for in-depth evaluation of individuals. 

improve computer interface and program- its counter intuitive adn unecessarily complex 
initial use should be on a smaller scale, the individual IPT level, with opportunities for interaction 
and team feedback 

It would be nice if there was a way to take out the personal criticism and just use the work 
related criticism. 

Leave it up to the supervisors to eliminate jealousy/competition/friends among responses. Team 
evals make sense, not individuals. 

Leave peer evals at the discretion of individual teams. 

List Prime Contractor employees by name. 

Maybe add a few personality type questions. 

More detailed survey. 

More individuals 
NA 

No comments right now. 

Not provide it to mgmt so you get more reliable data. 

Not sure 

not sure - not a true sample, 
once a year is frequent enough 

Probably nothing. People just have to take the time to do it. 
seems to pretty good now-cannot think of improvements 

Some of the teams are so small that it is obvious who wrote the comments, etc. This makes it a 
little disconcerting and makes it difficult to be as honest and forthwright as I'd like to be. 

The process is not broken. I responded "Neutral" because some of the comments I received last 
year were direct and helpful while other comments were ambiguous. To improve the process, 
then, would be to improve each evaluators approach to making comments, i.e. training for 
complete, unambiguous responses. 

The process seems a little flawed, blind feedback is generally negative since its easier to give 
(no confrontation). 

A listing of responsibilties (with % time) would be helpful in providing fair feedback. 

Add a category between sometimes on/sometimes off track and almost always there. 

Better evaluation choices 
feedback 

I don’t know how but make it more anonymous. 

I feel that that evaluation process should be on a 1 2 month timeframe and an overall cumulation 
of all inputs at the end. 

I received a lot of non-constructive sarcastic criticism during the last evaluation process. It 
would be appreciated if some of the comments could be filtered. 

Include contractor team members to be evaluated 

Lessen the time allowed, forcing people into deadline for doing this. 

not always have the same mulitple choice answers 



-establish a criteria for these evaluation -too much is left to interpretation 

-people can use this who have a ax to grind 

NOTHING 

Provide some sort of incentive to submit evaluations. Use a Macintosh computer. 

Reduce time required. 

Separate questions for managers and supervisors since we employees do not receive work from 
them. 

Share the generic information with rest of team. 

Show the metrics by which they are being evaluated. Then use the metrics to rate the 

performance. 

simplify if possible 

So long as it was not used for performance appraisals, I would like to have evaluations from all 
team members, not just Civil Servants. 

This is needed, but is time consuming. Why should i complain for an hour. Guess my frustration 
is not always knowing how to articulate in a constructive way what i want to get across. 3 
months ago my opinion might be different than what it is today. I can't carry stuff I want to say 
around for months and of course your opinion changes dependent on how another person has 
affected you. Overall, still a good process. 

This system is slow to use. Requires too much slow scrolling around. Takes much too long for 
such a few questions. 

Use information only for self improvement 
Use information only for self improvement 

Do not give it to supervisors. I feel very reluctant to provide candid comments 
to a co-worker if these comments are being given to his supervisor. This in my view 
depreciates the value of this evaluation. 

Don't make it any longer. 

Dream up a way to make it entirely painless. 

I don't really have any suggestions, 
n/a 

Open ended questions (or justification for answers to the numbered questions) provide the most 
valuable feedback. I know people hate to provide written feedback but that type is much more 
useful than mean scores/standard deviation/etc. 
should not pressure people into doing this 

Should sort out internal/external team inputs , supervisor/non supervisor managers 
feedback, subordinate, etc.. 

The application used need more modification to make for a more intuitive process, that is 
actions by the users such buttons to push, etc... a good start and progress from last year... still 
needs resources spent to further refine 
Too long. 

Use workstations with real mice! 

Show feedback resulting from this survey. 



Team Evaluation Process 


3. Do you feel that teams will get useful information from this process? 

1 . Strongly Disagree (Not at all useful) 1 1 4.80% 

2. Disagree (Not Useful) 12 5.24% 

3. Neutral (Some of this info, is useful) 99 43.23% 

4. Agree (Useful) 89 38.86% 

5. Strongly Agree (This info, is vey useful) 18 7.86% 

4. What could we do to improve the process of evaluating teams? 

Get rid of it. 

Not much. These things are always just a 'feel good' to make the managers happy. 

It really means nothing to me. But my manager said we 'had' to do this. 

See the answer to question 2. 

Stop! It only creates action items that don't take care of the real concerns. Let the teams figure 
out what they do well and poorly for themselves 
if they don't, someone will come up and tell them. 

Stop! It only creates useless action items that do not address the real concerns. 

Be more specific in response requirements, 
get the feedback to the troops! 

look at what they are supposed to do(i.e. team metrics) and how well they do it. 

Add to the evaluation what the groundrules are. I think that my team strives to do the best it 
can, however, present schedules are not allowing us to do this. Being schedule driven creates a 
hazard within the team in that people make more assumptions and tend to communicate less 
(e-mail is not read and becomes obe, etc). The team lead becomes another pair of thinking 
hands that become distracted from team wellfare. 

The set of groundrules needs to recognize items as decribed above and give the reviewer a set 
of criteria thorugh which the evaluation becomes more pertinent to the job at hand. 

Don’t really understand the benefit of this process. As I recall, the fundamental concept of IPT's 
and AIT s was that each team would be active in evaluating members' performance. How does 
this process differ from or add to that? 

Evaluate metrics based on responsibilities 
Feel that this is a good start 
get all members involved 
Get my team on the list. 

Have minimum of stations available at all times to make data collection a continuous process. 
Sincerely using the information collected. 

Have each evaluator provide unambiguous comments. 

I don’t know. I wish I did. this is probably the best way, but I'm still a little uncomfortable with 
it. 

make comments to answers mandatory for answers that rate less that a (3)... if the rating is 
good then there are no commenst required, why: we are in a teaming environment and should 
be willing to help others solve the problems we point out!! ust throwing stones is easy... don't 
let folks throw stones unless they are without sin (which none of us are) or can offer s olution. 
Make it available year round and allow individuals to schedule when they evaluate. Don't limit to 
once a year. 

make sure everyone participates. 

More detailed survey, 
not sure - not a true sample. 

Observation. It s interesting that management wants to evaluate team performance yet hires 
"old-school" managers (who do not necessarily buy-in to the team concept) to lead key teams 
such as the VIPT and CM AIT. Mgmt.'s views on teaming certainly impact how well NASA and 



Boeing individuals function together as a team. Hopefully, all newly hired mgrs will be required 
to take IPT training. Re: improving process, it might be more useful to evaluate teams after the 
reorganizations in O&U and Veh. Ofc. have been completed and stabilized. (This assumes that 
the BMO reorganization has indeed stabilized.) 
once a year is frequent enough 

Probably nothing. People just have to take the time to give a reasonable evaluation. 

Provide capability to evaluat teams in groups. For example, I would have liked to evaluate GN&C 
AIT and GN&C analysis together since they have a significant overlap in personnel and 
actiivities. 

This is obviously only useful if the team lead shares the results with the team. I don't know if 
many will. Additionally, thanks to the constant reorganization of the Station Program, I think 
this evaluation may be OBE in many cases. 

Too many teams to get good team to team comments. Need another technique like Valintines 
etc. 

- ask more detailed questions 

- provide a few more instructions 

- it was helpful having assistance present 

Add more questions that are more detailed and specific. This should be quicker (questions acts 
as prompts) and more informative to the teams, 
better focused answers to go with question 
Continue this process yearly. 

Do this twice a year. 

Ensure that team leads read this information, and actively try to implement suggestions. 

Ensure the recommendations are forwarded to the appropriate IPTs. 

I think the information would be very valuable, if it could be quantified (not to an individual 
level, but at least to an IPT or group level) 

EXAMPLE: I could better evaluate the responses and make improvements if I knew the 
information originated within the team vs. external to it (the same response could result in 
different action depending on source) 

Let the team pull together a team to address inputs (positive/negative) for corrective action. 

List of team responsibilities, and events which affected team's activities. 

MAKE THE RESPONSES TO THE SURVEY MORE VISIBLE, ESPECIALLY WHERE SOME KIND OF 
PROACTIVE CHANGE IS INITIATED. I’VE SEEN TOO MANY SURVEYS TAKEN, SEVERAL 
MONTHS AFTER WHICH, THE RESPONDENTS ASK "WHAT EVER BECAME OF THAT SURVEY 
WE TOOK?" 

No recommendations to date. 

NOTHING 

See comment on 2. 

The process does not provide for addressing general comments about the program or program 
management (ie it is my opinion that there is at best a very weak process for integrating the 
system design / program.) It was not clear how to inject this into the evaluation process. 

Twice a year process is good so that teams can use the information to put together action plans 
for improvement if necessary and have feedback again in a few months to see if they've been 
successful. Also, providing evaluators with reminders of who they evaluted would help us in 6 
mos to make sure we evaluate the same groups/individuals 

When data has been reduced, share statistical information across ISS Program. Response to 
these kinds of surveys must be shared with the participants in order to make them useful. 

Finally, a committment by management to respond to suggestions for improvement is very 
important. Unless some action results, there will be even less participation in the next one. 

Would like ability to give feedback to contractors on individual performance 

.Make the teams deal with the problems highlighted in the survey 

Process seems good with computers and also electronically sending it or mailing it. 

I like the TEAM EVALUATIONS much better than individual evaluations. Many of the challenges 
faced in this program are due to team interaction (not individuals). Need to make sure the team 



list is accurate and complete. For example, the EVR AIT wasn't listed, but an EVR IPT (of which 

I’ve never heard of) was-l assume the EVR IPT was supposed to be the EVR AIT (although the 

NASA lead was different.) 

none 

n/a 

People feel that because you enter your social security number that privacy is not maintained. 
Something should be done to help alleviate those fears. Say, have the developers named 
explicitly with statements stating what the social security numbers are used for. Even go so far 
as state the algorithms/logic flows that are employed. Actions taken such as this will help ease 
the concern. Until those fears are addressed the results will never be accurate 
Provide a hard copy of the team structure for new comers to the program who might be 
providing and evaluation for the first time. 



APPENDIX B-8c 

Session Evaluation Feedback (Team Process) 




SESSION EVALUATION FEEDBACK (SPRING '96) 

Number of Rators:412 

Number of Responses: 247 59.95% 


Team Evaluation Process 


1 . Do you feel you get useful information about yourself from this feedback? 


1 . Strongly Disagree (Not at all useful) 

15 

6.73% 

2. Disagree (Not Useful) 

15 

6.73% 

3. Neutral (Some of this info, is useful) 

111 

49.78% 

4. Agree (Useful) 

60 

26.91% 

5. Strongly Agree (This info, is vey useful) 

22 

9.87% 


2. What could we do to improve the process of evaluating individuals? 

Get rid of it. 

I matrix into the team. In this evaluation process, all I can do is input. Why are we only half-part 
of the process when we’re fully part of the team? 

If I have a specific beef, I take it up the team management structure, land I have). The 
fill-in-the-blank routine makes it hard to communicate a specific message or suggestion. Only 
the free-form entry really seems worth our while to enter. 

Not useful to me, I am a Contract engineer and am not included in this process aidividually. 
Suggets you add contractors to the individual appraisal process 

Since, with all of your assurances I still feel that my answers can be tracked down to me and I 
am not about to risk my job I won’t be really accurate. 

there are no controls or filtering. This program creates such stress that you are likely to get 
folks who lash out because not only are they frustrated in general, but they also are forced to 
spend valuable time doing what is perceived to be a non-value added activity. I haven't heard 
one person say one thing positive about this process. Given that environment, the data is very 
suspect. 

An On-line form at my office PC 

et their supervisors rate them based on their productivity not on personal feelings which that 
individuals may have regarding a team leader. It is not possible to control someones personal 
feelings that may have been provocate to give a bad rating when you no nothing of the 
circumstances that may have caused a bad report. 

Limit the number of people each can and will evaluate. 

the best evaluation of an individual comes from his/her supervisor. Improve that process. 

There is not enough time to rate all the folks I work with. I have a feeling other evaluators have 
the same feeling and don't spend much time on my own evaluation. 

This type of reveiw can be dangerous. From comments, one can deduce who provided negative 
feedback. This can cause hardfeelings in the organization. I think focus groups where folks get 
together to work out problems are more productive. Team leads need to hold these type of 
"retreats with their team and take issues forward issues to management. Make this part of the 
leaders and supervisors performance plans. A standard template and report could be required. 
Add Boeing Prime individuals. 

Alter the format. 

do it only once per year so people can take adequate time to provide constructive criticism 
- twice a year means that people may rush through knowing that they can do it again in 6 mos 
Do not allow supervisors to evaluate their employees. This distorts the feedback. 

Don't make us do this in the middle of crises period, you may get better response 

Due to limited time, couldn't evaluate everyone I'd like. Would be great if this could be accessed 

from the PCs. 

Expand the question base. Perhaps have more questions and allow evaluator to select which 
questions he wants to answer for a specific individual. 



Go one level lower so that the subteams can evaluate thems within a team. 

Have a list of adjectives describing people and their behavior that one could select applicable 
traits and thus provide a word-picture of an individual. For instance, bright, dull, cooperative, 
reluctant, tardy, punctual, brusque, cordial, 

Have the personnel develop a list of ALL individuals they have interacted with and make that list 
mandatory that they evaluate everyone on the list. By doing this the responses that we get as 
feedback should average out the extreme responses. 

Have the team members create a list of all personnel they interact with. It should then be 
mandatory to evaluate everyone on your list (ie. at least everyone in your immediate office). This 
will average out the extreme responses and provide better feedback. 

I am not sure this is helpful for evaluating individuals because the results can be easily 
misleading if someone has an axe to grind against someone. 

I thning this is a good system for a cursory level evaluation of a large group of people. Not sure 
of the effectiveness for in-depth evaluation of individuals. 

improve computer interface and program- its counter intuitive adn unecessarily complex 
initial use should be on a smaller scale, the individual IPT level, with opportunities for interaction 
and team feedback 

It would be nice if there was a way to take out the personal criticism and just use the work 
related criticism. 

Leave it up to the supervisors to eliminate jealousy/competition/friends among responses. Team 
evals make sense, not individuals. 

Leave peer evals at the discretion of individual teams. 

List Prime Contractor employees by name. 

Maybe add a few personality type questions. 

More detailed survey. 

More individuals 
NA 

No comments right now. 

Not provide it to mgmt so you get more reliable data. 

Not sure 

not sure - not a true sample, 
once a year is frequent enough 

Probably nothing. People just have to take the time to do it. 
seems to pretty good now-cannot think of improvements 

Some of the teams are so small that it is obvious who wrote the comments, etc. This makes it a 
little disconcerting and makes it difficult to be as honest and forthwright as I'd like to be. 

The process is not broken. I responded "Neutral" because some of the comments I received last 
year were direct and helpful while other comments were ambiguous. To improve the process, 
then, would be to improve each evaluators approach to making comments, i.e. training for 
complete, unambiguous responses. 

The process seems a little flawed, blind feedback is generally negative since its easier to give 
(no confrontation). 

A listing of responsibilties (with % time) would be helpful in providing fair feedback. 

Add a category between sometimes on/sometimes off track and almost always there. 

Better evaluation choices 
feedback 

I don't know how but make it more anonymous. 

I feel that that evaluation process should be on a 12 month timeframe and an overall cumulation 
of all inputs at the end. 

I received a lot of non-constructive sarcastic criticism during the last evaluation process. It 
would be appreciated if some of the comments could be filtered. 

Include contractor team members to be evaluated 

Lessen the time allowed, forcing people into deadline for doing this. 

not always have the same mulitple choice answers 



-establish a criteria for these evaluation -too much is left to interpretation 

-people can use this who have a ax to grind 

NOTHING 

Provide some sort of incentive to submit evaluations. Use a Macintosh computer. 

Reduce time required. 

Separate questions for managers and supervisors since we employees do not receive work from 
them. 

Share the generic information with rest of team. 

Show the metrics by which they are being evaluated. Then use the metrics to rate the 

performance. 

simplify if possible 

So long as it was not used for performance appraisals, I would like to have evaluations from all 
team members, not just Civil Servants. 

This is needed, but is time consuming. Why should i complain for an hour. Guess my frustration 
is not always knowing how to articulate in a constructive way what i want to get across. 3 
months ago my opinion might be different than what it is today. I can't carry stuff I want to say 
around for months and of course your opinion changes dependent on how another person has 
affected you. Overall, still a good process. 

This system is slow to use. Requires too much slow scrolling around. Takes much too long for 
such a few questions. 

Use information only for self improvement 
Use information only for self improvement 

Do not give it to supervisors. I feel very reluctant to provide candid comments 
to a co-worker if these comments are being given to his supervisor. This in my view 
depreciates the value of this evaluation. 

Don't make it any longer. 

Dream up a way to make it entirely painless. 

I don't really have any suggestions, 
n/a 

Open ended questions (or justification for answers to the numbered questions) provide the most 
valuable feedback. I know people hate to provide written feedback but that type is much more 
useful than mean scores/standard deviation/etc. 
should not pressure people into doing this 

Should sort out internal/external team inputs , supervisor/non supervisor managers 
feedback, subordinate, etc.. 

The application used need more modification to make for a more intuitive process, that is 
actions by the users such buttons to push, etc... a good start and progress from last year... still 
needs resources spent to further refine 
Too long. 

Use workstations with real mice! 

Show feedback resulting from this survey. 


Team Evaluation Process 


3. Do you feel that teams will get useful information from this process? 

1 . Strongly Disagree (Not at all useful) 1 1 4.80% 

2. Disagree (Not Useful) 12 5.24% 

3. Neutral (Some of this info, is useful) 99 43.23% 

4. Agree (Useful) 89 38.86% 

5. Strongly Agree (This info, is vey useful) 18 7.86% 

4. What could we do to improve the process of evaluating teams? 

Get rid of it. 

Not much. These things are always just a 'feel good' to make the managers happy. 

It really means nothing to me. But my manager said we 'had' to do this. 

See the answer to question 2. 

Stop! It only creates action items that don't take care of the real concerns. Let the teams figure 
out what they do well and poorly for themselves 
if they don't, someone will come up and tell them. 

Stop! It only creates useless action items that do not address the real concerns. 

Be more specific in response requirements, 
get the feedback to the troops! 

look at what they are supposed to do(i.e. team metrics) and how well they do it. 

Add to the evaluation what the groundrules are. I think that my team strives to do the best it 
can, however, present schedules are not allowing us to do this. Being schedule driven creates a 
hazard within the team in that people make more assumptions and tend to communicate less 
(e-mail is not read and becomes obe, etc). The team lead becomes another pair of thinking 
hands that become distracted from team wellfare. 

The set of groundrules needs to recognize items as decribed above and give the reviewer a set 
of criteria thorugh which the evaluation becomes more pertinent to the job at hand. 

Don't really understand the benefit of this process. As I recall, the fundamental concept of IPT's 
and AIT's was that each team would be active in evaluating members' performance. How does 
this process differ from or add to that? 

Evaluate metrics based on responsibilities 
Feel that this is a good start 
get all members involved 
Get my team on the list. 

Have minimum of stations available at all times to make data collection a continuous process. 
Sincerely using the information collected. 

Have each evaluator provide unambiguous comments. 

I don’t know. I wish I did. this is probably the best way, but I'm still a little uncomfortable with 
it. 

make comments to answers mandatory for answers that rate less that a (3)... if the rating is 
good then there are no commenst required, why: we are in a teaming environment and should 
be willing to help others solve the problems we point out!! ust throwing stones is easy... don't 
let folks throw stones unless they are without sin (which none of us are) or can offer s olution. 
Make it available year round and allow individuals to schedule when they evaluate. Don't limit to 
once a year. 

make sure everyone participates. 

More detailed survey, 
not sure - not a true sample. 

Observation: It's interesting that management wants to evaluate team performance yet hires 
"old-school" managers (who do not necessarily buy-in to the team concept) to lead key teams 
such as the VIPT and CM AIT. Mgmt.’s views on teaming certainly impact how well NASA and 



Boeing individuals function together as a team. Hopefully, all newly hired mgrs will be required 
to take IPT training. Re: improving process, it might be more useful to evaluate teams after the 
reorganizations in O&U and Veh. Ofc. have been completed and stabilized. (This assumes that 
the BMO reorganization has indeed stabilized.) 
once a year is frequent enough 

Probably nothing. People just have to take the time to give a reasonable evaluation. 

Provide capability to evaluat teams in groups. For example, I would have liked to evaluate GN&C 
AIT and GN&C analysis together since they have a significant overlap in personnel and 
actiivities. 

This is obviously only useful if the team lead shares the results with the team. I don't know if 
many will. Additionally, thanks to the constant reorganization of the Station Program, I think 
this evaluation may be OBE in many cases. 

Too many teams to get good team to team comments. Need another technique like Valintines 
etc. 

- ask more detailed questions 

- provide a few more instructions 

- it was helpful having assistance present 

Add more questions that are more detailed and specific. This should be quicker (questions acts 
as prompts) and more informative to the teams, 
better focused answers to go with question 
Continue this process yearly. 

Do this twice a year. 

Ensure that team leads read this information, and actively try to implement suggestions. 

Ensure the recommendations are forwarded to the appropriate IPTs. 

I think the information would be very valuable, if it could be quantified (not to an individual 
level, but at least to an IPT or group level) 

EXAMPLE: I could better evaluate the responses and make improvements if I knew the 
information originated within the team vs. external to it (the same response could result in 
different action depending on source) 

Let the team pull together a team to address inputs (positive/negative) for corrective action. 

List of team responsibilities, and events which affected team's activities. 

MAKE THE RESPONSES TO THE SURVEY MORE VISIBLE, ESPECIALLY WHERE SOME KIND OF 
PROACTIVE CHANGE IS INITIATED. I'VE SEEN TOO MANY SURVEYS TAKEN, SEVERAL 
MONTHS AFTER WHICH, THE RESPONDENTS ASK "WHAT EVER BECAME OF THAT SURVEY 
WE TOOK?" 

No recommendations to date. 

NOTHING 

See comment on 2. 

The process does not provide for addressing general comments about the program or program 
management (ie it is my opinion that there is at best a very weak process for integrating the 
system design / program.) It was not clear how to inject this into the evaluation process. 

Twice a year process is good so that teams can use the information to put together action plans 
for improvement if necessary and have feedback again in a few months to see if they've been 
successful. Also, providing evaluators with reminders of who they evaluted would help us in 6 
mos to make sure we evaluate the same groups/individuals 

When data has been reduced, share statistical information across ISS Program. Response to 
these kinds of surveys must be shared with the participants in order to make them useful. 

Finally, a committment by management to respond to suggestions for improvement is very 
important. Unless some action results, there will be even less participation in the next one. 

Would like ability to give feedback to contractors on individual performance 

.Make the teams deal with the problems highlighted in the survey 

Process seems good with computers and also electronically sending it or mailing it. 

I like the TEAM EVALUATIONS much better than individual evaluations. Many of the challenges 
faced in this program are due to team interaction (not individuals). Need to make sure the team 


list is accurate and complete. For example, the EVR AIT wasn't listed, but an EVR IPT (of which 

I've never heard of) was--l assume the EVR IPT was supposed to be the EVR AIT (although the 

NASA lead was different.) 

none 

n/a 

People feel that because you enter your social security number that privacy is not maintained. 
Something should be done to help alleviate those fears. Say, have the developers named 
explicitly with statements stating what the social security numbers are used for. Even go so far 
as state the algorithms/logic flows that are employed. Actions taken such as this will help ease 
the concern. Until those fears are addressed the results will never be accurate 
Provide a hard copy of the team structure for new comers to the program who might be 
providing and evaluation for the first time. 



APPENDIX C 

Second Peer Evaluation Schedule 



Original Schedule forIR13.A 

The timetable, subject to change due to furloughs of the Federal Government, is as 
follows. 

December 11 - 15: Meet with appropriate JSC personnel to discuss needs and options; 
create Lotus views and forms templates and start developing those templates 

December 18-22: Receive the final version of the set of survey questions from JSC; 
develop Lotus question forms, output forms and statistics forms and statistics forms 
views 

December 26 - 29: Receive data from JSC and import data into Lotus; create security 
forms Lotus views and begin security testing 

January 2-5: Continue with security programming development and Lotus forms and 
views; install applications on the portable computer system; test the demonstration; 
perform a test demo on site at JSC on 1/5/96 

January 8-12: Make any necessary modifications as requested as a result of the demo; 
create poster instruction “help sheets” for the room where the survey will be held, set up 
the survey site at JSC and on 1/10/96 perform a final demo at JSC and install the survey 
on site. 

January 16 - February 19 Test and finalize survey and check for system stability 

January 22 - February 9 Conduct survey 

February 9-23 Analyze Data, printout survey, backup data 

February 23 - May 31 Train student on Lotus Notes and explore distributed possibilities 
of Lotus Notes and other groupware software 


Step 1: 


Appendix One: Portable GSS Training Support Handbook 


• Unpack computers from (nine notebook workstations and one server) cases and 
sequentially arrange in a row. 


Step 2; 

Server - 

• Open left side door to access the system’s PCMCIA socket. 

• Insert PCMCIA ethernet card (facing up). 

• Connect the patch cable’s Honda connector to the ethernet card. 

• Connect the 10Base-2 coax cable and the T-connector to the patch cable; this 
10Base-2 cable should have a RED terminator on each end. 

• Connect power supply to the back of the server. 


Step 3: 


Workstation Connection - 

Open left side door side to access system’s PCMCIA socket. 

The PCMCIA ethernet card should already been installed in the socket. 

Connect the Honda connectors to the card. 

Repeat for all laptops to be used. 

Interconnect all T-connectors with the thin net cables (black cables) to form a 
configuration as shown in the following figure: 

Terminator T-Connector 

/ Ethernet Cabling jf 
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Server Workstations 

Remove one of the red terminator from the server and attach it to the last 
workstation of the network. 


Step 4: 

• Attach power supplies to the rear of all computers. 

Step 5: 

• Connect panel book - LCD panel to the video port in the facilitator terminal. 

Step 6: 

• Turn the power on in the server. 

• At opening menu (only if system has been reconfigured for Lotus Notes in 
previous session), select 

“1. Reconfigure the system to run Novell” 

• Start Novell server. 


l 


Step 7: 


• Turn the power on in work stations. 

• At opening menu (only if system has been reconfigured for Lotus Notes in 
previous session), select 

“1. Reconfigure the system to run Novell” 

• Start Windows for Workgroups. Username for Workgroups is TERMINAL X 
where X is from 1 to 10, and the password is teamwork. 

Step 8; 

Software setup- 

• Start group systems for windows. 

• Username for facilitator terminal is “leader”, and omit password (blank). 

• Username for user terminals are sequential numbers from 1 through 8, and 
omit password (blank). 

Step 9: 

Shut down process for workstations- 

• Stop all participants from the facilitator terminal. 

• Release participant ready screen from the facilitator terminal, (optionally, 
press CTRL-HOME at each terminal). 

• Exit GroupSystems from the leader terminal. 

• Exit Windows. 

• Logout all workstations from the server. 

• Turn power off at all workstations. 

• Disconnect and pack panel book - LCD panel. 

Step 10: 

Shut down process for server - 

• Type “Down” at the server to shut down the server. 

• Turn power off at the server. 

Step 11: 

Packing system for transportation (Server) - 

• Disconnect power supply at the back of the server lap top. 

• Disconnect ethernet cable attached to the workstation, and connect the two red 
terminators to both ends of the T-connector. 

• Remove ethernet card and put in carrying case. 

• Close ethernet socket door. 

• Pack computer and power supply in carrying case. 

Step 12: 

Packing system for transportation (workstation) - 

• Disconnect all T-connectors attached to the thin net cables (black cables). 

• Repeat for all lap tops used. 

• Disconnect the ethernet connectors from the card (do not remove ethernet 
card). 
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Close ethernet socket door. 

Pack computer and power supply in carrying case. 

Remove red terminator from the server and attach the server to the end of the 
network. 

Connect the removed terminator described above to the other end of the cable. 



Appendix Two: Introduction for Participants Using GrnupSvstems for Windows 


The facilitator will read the appropriate sections of the following script. As the script is 
read the participants will see screen views of the tools projected on the front central 
projection screen in the meeting room. 


Group Systems for windows consists of 5 tools which help meeting participants diverge 
and/or converge information. The 5 tools are: 

1 . Electronic Brainstorming 

2. Categorizer 

3. Topic Commenter 

4. Group Outliner 

5. Vote 

The 5 tools represent two types of functional activities. 

1 . Idea Generating/Gathering. 

2. Decision Making 

1. Idea generating/gathering (Diverging) Tools 


1. 1 Electronic Brainstorming 

Electronic Brainstorming is an idea-generating tool that allows participants to share ideas 
simultaneously and anonymously on a specific question posed to the group. Using their 
keyboards, participants engage in an electronic discussion of a question; they are able to 
contribute ideas at the same time with no loss of information, unlike traditional 
brainstorming sessions. 

(Slide 1) The leader enters a question or issue to provide a stimulus for participants. In 
this case the question is “what does the electronic brainstorming participant screen look 
like?” The electronic equivalent of sheets of paper are distributed, one per participant. 
After a participant enters an idea, the discussion sheet is exchanged for another; this sheet 
may display an idea entered by another participant. (Slide 2) The participant can 
comment on the existing response or can begin a new line of thought. As the activity 
progresses, each discussion sheet gathers an assortment of ideas which can spark 
responses in strikingly different directions. 

(Slide 3) Participant ideas can be categorized by comments using keywords. When you 
add a keyword, the uncategorized comments are searched for occurrences of the keyword. 
If the keyword is found, the comment is taken from the uncategorized comments and 
grouped under the keyword. (Slide 4) 

A dialog on the electronic sheets of paper is demonstrated at this point. (Slide 5) 
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Electronic Brainstorming - Slide 2 
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Electronic Brainstorming - Slide 4 




Electronic Brainstorming - Slide 5 
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1.2 Categorizer 

(Slide 1) The Categorizer helps groups in three common group activities: 

1. Generating lists of ideas 

2. Brainstorming comments which elaborate on or support the ideas 

3. Creating categories for the ideas. 

The categorizer is a tool that can be used in a variety of ways. 

(Slide 2) In the list-building mode , each participant enters ideas related to the issue being 
considered by the group. These ideas are added directly to the group list. It is possible to 
edit the group list to consolidate similar ideas, make it more manageable and add 
comments to any idea on the list. The leader also can bring in a list from another activity. 

(Slide 3) In categorizing mode, participants can create categories which relate to the ideas 
on a list that they built, or which was shifted from another GroupSystems activity. They 
then can copy the ideas to the appropriate category. (Slide 4) 

The participants also can work from the top down, by first entering or having the group 
create the categories, and then adding ideas and comments appropriate to each category. 
(Slide 5) 
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Gatcgorizer - Slide 3 


GroupSystems - Template - [Categorizer (Categorizer)] 
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Categorizer - Slide 4 
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Categorizer - Slide 5 
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1.3 Topic Commenter 
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1.4 Group Outliner 

Group Outliner helps groups generate and/or organize ideas. It is well suited to filling in 
the details of a business plan or research proposal when the general outline is known. 
Comments or details then can be quickly generated by participants. 

As an idea generating and information gathering tool, Group Outliner functions much like 
Topic Commenter, in which participants enter comments about topics on a list. (Slide 1 ) 
Unlike Topic Commenter, however, each Group Outliner topic can have subtopics 
arranged in the familiar hierarchical structure of an outline. (Slide 2) Thus, the idea 
generation process is more highly structured than that of either Electronic Brainstorming 
or Topic Commenter. 

A topic is any item on the outline. Topics that branch off from another topic (which is 
the parent topic) are referred to as subtopics. Therefore, an outline item can be 
considered either a topic or a subtopic, depending upon the point of reference. 

The same is true of levels and sublevels. A level is a topic, or list of topics, linked by a 
vertical line on the outline. A sublevel is a subtopic, or list of subtopics, linked by a 
vertical line. 

Levels are referred to by numbers in the topics editor. The main topic of the outline, or 
session name, is level 1. The other levels are numbered to level 8 according to their place 
in the hierarchy. (Slide 3) 

Starting with the idea under discussion, or from a partial outline, participants suggest 
topics and subtopics to be organized in outline form. Participants can then enter 
comments about each topic and subtopic. (Slide 4) 
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GroupSystems - Template - [Group Outliner (Group Outliner)] 
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2. Decision Making (Converging) Tools 

The Vote tool represents the decision making part of Group Systems. 

These are demonstrated for each voting tool by the “nView” slide show presentation 
diskettes included with this report. 

All the vote tools follow a similar pattern. 

Step 1: Create a list of vote items. (Slide 1 ) 

Step 2: Add supporting comments to the list of vote items. ( Slide 2 ) 

Step 3: The technical facilitator shifts the mode from a list building to a vote 

mode, and the participants cast the vote. (Slide 3) 

Step 4: The vote spread and statistical results are viewed by the group. (Slide 4) 

Step 5: The vote graph results are viewed by the group. (Slide 5) 

The slides that follow provide a representative view of how all the vote tools will appear. 
For specific views, run the diskettes that accompany this report. 
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2.1 Vote 

Vote offers a variety of methods for polling group opinion, on one or many issues. You 
can select from the following eight voting methods: 

1. Rank Order 

A voting method in which you reorganize a list of ballot items by dragging items to 
the desired position. 

2. 10-Point Scale 

A voting method in which you rate each item in a list using an integer scale from one 
(low) to ten (high). 

3. Multiple Selection 

A voting method in which responses are selected from a list of ballot items. You can 
choose one or several items, based on the number of choices the leader specifies 

4. Yes/No 

A voting method in which each item on the list is a question to which you respond 
“Yes” or “No”. 

5. True/False 

A voting method in which each item on the list is a question to which you respond 
“True” or “False”. 

6. Agree/Disagree (5-point ) 

A voting method in which you indicate the extent of your agreement (Strongly Agree, 
Agree, ...) with each item on the list. In this method Strongly Agree is given a weight 
of 5, Agree is 4, etc. 

7. Agree/Disagree (4-point) 

A voting method in which you indicate the extent of your agreement (Strongly Agree, 
Agree, ...) with each item on the list. In this method Strongly Agree is given a weight 
of 4, Agree is 3, etc. 

8. Custom Method 

A voting method that the leader can customize to meet the needs of the group. 

As the name implies. Vote is an evaluation tool, capable of providing the basis for a 
group decision. This tool also is commonly used to determine the degree of group 
consensus. Vote can be utilized periodically following discussions to determine whether 
the group consensus has shifted. Another way to use Vote is as an information-gathering 
tool in much the same way as an opinion poll. 

To use Vote, the leader enters the list of ballot items and selects a voting method. You 
- can add supporting comments for "ballot items to clarify the meaning or significance of the 
items. You can add supporting comments to any ballot item on the list, or view the 
existing comments about an item. Comments can be added only during list-building 
mode but can be viewed at any time, even while in voting mode. 
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While you are in list-building mode, information markers guide you to comments you 
have not viewed. For each ballot item, the total number of comments and the number of 
new comments appear beside the ballot item name. 

Participants enter their choices according to the type of vote chosen. Results can be 
viewed in a vote spread matrix or graph. 

Ballots are collected automatically as they are cast. In voting mode, you can view the 
cumulative results of all votes in both graphical and statistical forms. If additional ballots 
are cast while you are viewing results, the results are updated automatically. Although 
this feature will be used primarily by the leader, he or she may give participants the 
option to view results as well, using the View Results group setting. When you choose 
View Results from the Vote menu, the “Vote Graph” window appears, overlaying the 
“Vote Spread” window, and providing “Vote Statistics”. 

The criterion by which results on both windows are arranged depends on the selected 
voting method. You can sort voting results by any of the statistics available on the Vote 
Spread window. 

1. Vote Graph Window 

The Vote Graph window displays the voting results in a horizontal bar graph. 

2. Vote Spread Window 

The Vote Spread window displays statistical information about the voting 
results. The statistics displayed in the Vote Spread window depend on the voting 
method. 

A voting matrix is available for Rank Order, 10-Point Scale, Agree/Disagree, and 
Custom Method votes. The rows list the ballot items and the columns display the 
number of votes for each possible ranking. 

Columns labeled with a percentage sign, such as Yes% or False%, display the 
percentage of the vote. These appear only for Yes/No and True/False votes. 


Total displays the total of the votes. The precise meaning of total varies according to 
the voting method: 

• For Rank Order, total is the rank sum. 

• For Multiple Selection, total is the number of times the item was chosen. 

• For Yes/No and True/False, total is the number of yes or true votes. 

• For the remaining voting methods, total is the weighted sum. 

Mean indicates the average ranking for each item. Usually this produces the same 
order as the total. Abstentions, however, affect the total but not the mean, and so the 
two sorting orders may differ. 
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STD indicates the standard deviation for each item that has a mean displayed. N 
indicates the number of participants who voted, excluding bypasses. “You” is an 
optional column that appears only if the Voter Comparison group setting is enabled. 
It displays how you voted on the ballot item. 

Zval is another optional column that appears if Voter Comparison is turned on for 
Rank Order, 10-Point Scale, Agree/Disagree, or Custom Method votes. 

The total number of voters (N) is displayed at the bottom of the window. If a ballot 
item is too long to fit in the vote matrix, you can select it and view the full text in the 
Selected row area. 

Rank Order votes display two additional statistics at the bottom of the window: 

• Group consensus measures the level of agreement among voters. 

• Your agreement with the group results measures how closely your votes match 
the group results. 

3. Vote Statistics 

The following statistics are used on the Vote Spread window: 

Rank sum, which determines the "total" in Rank Order votes, is calculated by 
multiplying the number of votes an issue receives by the weight of its position. Unlike 
10-Point Scale weights, however, a ranking in the first position (i.e., the top of the list) 
results in the highest weight. 

For example, when ranking a list of three items, the first position has the weight of three, 
the second has two, and so on. If two participants rank an item in first place, then the 
rank sum for that item is six. The rank sum is thus not directly related to the mean, 
because the mean is the average ranking. 

Weighted sum, which determines the "total" in 10-Point Scale, Agree/Disagree, and 
Custom Method votes, gives a weight to each votes. The weight given depends on the 
voting method: 

• For 10-Point Scale, the weighted sum is computed by multiplying the number of 
votes an issue receives by the weight of its ranking. 

For example, a ranking of 10 is given a weight of 10. 

• For Agree/Disagree (5-point), a rating of Strongly Agree is given a weight of 5, 
Agree is 4, etc. 

• For Agree/Disagree (4-point), a rating of Strongly Agree is given a weight of 4, 
Agree is 3, etc. 

• For Custom Method votes, the weight given each choice is determined by the 
corresponding value entered in the Custom Method dialog box. 

Standard deviation is a measure of the distribution of responses over the total number of 
responses. If all participants give an item the same rank, the standard deviation is zero. 
Therefore, low standard deviation indicates agreement among participants, while high 
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standard deviation indicates disagreement. On a standard bell-shaped curve, 95% of 
responses fall within two standard deviations of the mean. 

Zval is short for Z value, a statistic representing the number of standard deviations the 
participant s ranking differs from the group mean, either above (positive number) or 
below (negative number), assuming a normal distribution. Approximately two-thirds of 
the group falls within one Z value from the group mean. Any Z value over 2 or below -2 
can indicate a significant difference between the individual and group votes. The Z value 
is displayed if the Voter Comparison group setting is turned on for Rank Order, 10-Point 
Scale, Agree/Disagree, or Custom Method votes. 

Group consensus, which is available for Rank Order votes, uses a statistic called 
Kendall's Coefficient of Concordance to measure the level of agreement among voters. 
The formula for this statistic takes the amount of variation in rank sum voting and divides 
it by the total possible amount of variance. The closer to 1 .0 the group consensus figure 
is, the more homogeneous the group's ranking patterns. 

Your agreement with the group results, which is available for Rank Order votes, uses a 
statistic called Kendall's Tau Coefficient to measure how closely your votes match the 
group results. The figure for individuals can range from -1.0 and 1.0. The closer to 1.0 
the individual consensus figure is, the greater the agreement with the group. 

For example, if the group has a consensus rating of .2, and the individual consensus was 
.8, the group did not vote consistently but the participant's ranking closely matched the 
group average. 

Advanced features allow participants to view the results of their vote in comparison to the 
group's voting patterns. 1 


1 Much of the text used in this script was adopted from the help menu provided with the 
GroupSystems for Windows software. 
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Appendix Three; Matrix of JSC Meeting Types 


GROUP ISSUE 

Commitment to goals, 
purpose, objectives 


Communication problems/ 
Conflict between people 

Not enough 

participation 


One person 
dominates 


Conflict around issues 


Creativity/ People stuck 
“In the box” 


Focus 


Face-to-Face Facilitation Solution 


• Describe perceived lack of interest by group. 

• Ask others to comment on the perceived lack of interest. 

• Solicit suggestions for improving participation of group. 


• Process check — tell group what you see. 

• Check for adequacy of agenda — make sure group is addressing real 
task at hand. 

• Ask quiet people for suggestions. 

• Ask for others to comment on what the dominating individual is 
saying. 

• Thank dominant individual for contributions than ask to hear from 
others. 


• Describe the conflict observed. 

• Ask group for solution to moving beyond perceived conflict. 

• Use a thumb up/thumb down method to check for agreement. 


• Comment on the perceived traditionalness of the ideas that are 
being put forth. 

• Encourage more nontraditional new ideas. 

• Specifically ask for outrageous solutions. 


• Describe the observed lack of focus on topic. 

• Ask group to come up with suggestion on how to focus discussion.. 

• Stop discussion and ask for group come to agreement on what focus 
should be. 
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GROUP ISSUE 

Commitment to goals, 
purpose, objectives 


Communication problems/ 
Conflict between people 

Not enough 

participation 


One person 
dominates 


Conflict around issues 


Creativity/ People stuck 
“In the box” 


Focus 


GDSS Solution 


• Process check on adequacy of agenda. 

• Use Electronic Brainstorming to solicit reasons for perceived 
lack of commitment — use results to open up discussion. 


• Use any GDSS tools to increase contributions. 


• Use any GDSS tools to equalize contributions. 


• Use comment tools to ask for potential solutions/elaborations. 

• Use Vote to poll for agreement — use results to start 
discuss about issue at hand. 


• Use Topic Commenter and specifically label topics as ideas 
that wouldn’t work (UN-ideas), new twists on our old ways, 
very traditional ideas, ideas that would rock the boat, etc... any 
labels that might encourage divergent creative thinking. 

• Use Electronic Brainstorming and encourage others to use 
other ideas and twist them around and resubmit. 


• Use Topic Commentor or Categorizer. Can ‘diffuse’ energy 
by commenting on many topics or gain agreement and focus by . „ 
agreeing on one topic generating comments. 
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GROUP ISSUE 

Face-to-Face Facilitation Solution 

Group Energy 

• Comment on perceived lack of energy in room. 

• Ask for suggestions for increasing energy. 

• Try a short break where people walk around and try to come up with 
ideas around the task at hand. 

Leadership 

Not enough- 
group drifting 

• Look for ways to reengage appointed leader of the group or ways to 
reach group consensus on group direction. 

• Process check; tell group what you see, suggest route, or ask leader 
directly to step up and give guidance. 

Too much - 
leader too strong 

• Look for ways to diffuse leadership that squelches ideas 
from the group. 

• Process check; tell group what you see — suggest new route or ask leadei 
directly to allow the group a few minutes to discuss the 

issues for themselves. 

Too many — 
how to solve “too 
many chiefs" 

problem 

• Look for ways to gain agreement on one particular course of action. 

• Process check; tell group what you see, suggest new route or ask for 
group to agree on one course of action. 

Organization 

* Organization should be less of a problem with the use of Group Support Systems 
because of the necessity of a preplanned agenda. 

Inadequate • Revisit purpose of meeting and use that information to construct 

agenda new agenda. Gain agreement from group on changes. 

• Take a short break and confer with group leader. 

Too much data 

• Nominal group technique to decide priority issues. 

• Gain agreement from group on how to deal with data at a later time. 
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GROUP ISSUE 


GSS Solution 


Group Energy 

• Try Electronic Brainstorming to collect ideas. 

• Try Topic Commenter to send out trivia question to bring 
energy level back up. 

Leadership 

Not enough- • Hold more extensive preplanning meetings. 

group drifting • Process check on adequacy of agenda. 


Too much — • Process check on adequacy of agenda 

leader too strong • Stronger reliance on GSS tools w/out as much talking. 

• Facilitator should direct attention to comments from group members & 
direct less to leader. 


Too many — • Process check on adequacy of agenda. 

how to solve “too • Stronger reliance on GSS tools w/out as much talking — try directing 

many chiefs ” attention toward data or get them to focus on specific 

problem ideas and gain consensus on those ideas. 

Organization 

* Organization should be less of a problem with the use of Group Support Systems 
because of the necessity of a preplanned agenda. 

Inadequate • Once there is an understanding of the appropriate changes to 

agenda make to the agenda, conduct the rest of the meeting in a 

face-to-face format or take a break and program a new agenda 
into the system. 

Too much data • Use a data reduction method like Vote, or use Categorizer to 
organize and/or create categories of comments. 
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GROUP ISSUE 


Face-to-Face Facilitation Solution 


Problem Solving/ Decision Making 

Too quick to • Question whether an adequate understanding of the problem exists. 

adopt solution • Ask for a devil’s advocate to counter current analysis of situation or 
to come up with reasons why solution wouldn’t work. 

• Ask for participants to take a different perspective for a few minutes. 


Time crunch 

• Remind group of agreed to ending time or comment on perceived 
slowness of progress. 

• Ask for suggestions on how to move along more quickly. 


GROUP ISSUE 


GSS Solution 


Problem Solving/ Decision Making 

Too quick to • Use Topic Commentor or Categorizer to better define 

adopt solution problems — ask for specific drawbacks of a solution 
suggested. 

• Use Electronic Brainstorming and ask everyone to play 
devil’s 

advocate. Use Vote to decide on any objections that really 
need to be discussed. 


Time crunch 

• Use Vote to gauge effectiveness of conversation. 

• Vote on importance of remaining agenda items - 
decide which to drop if possible. 



Appendix Four: Review of Sessions and Participant Feedback 

Session participants were asked their opinion about how they perceived the session 
they attended. A sample of their feedback is reproduced in this appendix. Spelling and 
punctuation were not corrected but names were deleted to preserve the anonymity of 
the group. The feedback questions were standardized a short time into the process. 

Meeting A 

Where did the process succeed? 

The facilitators did an excellent job — there is a lot of information to absorb. 

Strong facilitation is a must for leading sessions. Facilitators must be trained in the use of the technology 
and facilitation skills in general to promote effective use of the technology by the participants. 

It was my first exposure to group decision making software, and it was valuable in understanding the 
concept 

expands my thoughts as to what is possible 

Presented the potential of this technology to participants and the impact it can have on business 
productivity. The ability to generate discussions, ideas, make decisions- all these functions are greatly 
enhanced with this technological capability. 

More business owners/managers need exposure to this technology. 

A distributed model allowing different time/different place sessions would be much more valuable to [name 
deleted] than the meeting room mofel presented 

This is a wonderful lab. We appreciate your generosity and time in letting us use this and showing us this 
software. 

Beautiful facilities. Wonderful sense of cooperation between community, business and higher education. 
This would be an excellent tool to use in a brainstorming session. 

With only limited exposure it is difficult to have an indepth opinion. However, from what IVe seen the tool 
is limited in its application by requiring the same time/same place model. 

Great for brainstorming - would significantly reduce time in both generation of ideas and for future 
reference of discussion items generated. 

Where did the process fail? 

The more experience and training the user has with the tool the more valuable it is. With no forknowledge 
and limited time for explanation, some of the applications seemed less than intuitive in their operation 
our group needed a pre planning session 

Definitely needs to have a preview opportunity for participants to get a feel for being in a virtual office 
setting. Having the ability to key in at different times in a variety of different settings and locations will be 
a critical tool for enhancing business productivity. 

The facilitators were not quite up to speed on their knowledge of the software. 


Meeting B 

How clearly and accurately was the information discussed? 

Very clear 

THe program was easy to understand and use 

Excellent 

very clearly 

Discussion thorough in regard to use of pkg. 

Did not have any problem with understanding instructions or using the software. 


1 



Software is very easy to use, with the help of a facilitator. Explanations given were useful and not 
confusing. 

The concept of groupware was clearly discussed. I would like to have learned about some of the other 
software designs to understand and compare the competative advantages and disadvantages of this product. 
Clear 

This is an incredible tool which I could use to enhance my business capabilities . However, typing is not 
everyones strong suit therefore the key pad should be elimiminated whereever possible. 

How satisfied are you with the results of the meeting? 

I learned a lot today. 

The technology was exciting to use 
I appreciated the software and our progress today. 

The software use allowed us to participate simultaneously and allowed us to use time effectively. 

Exciting for a novice in most of the areas. 

Enjoyed using the technology. Didn’t know anything like theis existed. 

Well satisfied 

I was very satisfied and excited to be a part of this. 

As an organization and planning tool I am satisfied with the outcome. However I think that to maximize the 
demonstration process two days is necessary to really take advantage of the full potential use. By the time 
the learning curve and discussion had taken place the session was over. 

The results were not satisfactory to me, but the process was a joy, because it is so interactive. 

What changes would you make to the meeting process? 

More computers or two sessions 
Have everything in this Windows format 

Sharing sometimes caused complications and wasted time when classmembers raised questions about equity 
Allow for another program to be available to participants use while other class participants are finalizing 
their scoring or work. i.e. lets play poker.I 

More computers, of course. More time to ask questions about the capabilities of the software. 

Allow more time to work with program 
Other than too many people for computers available 
WOULD CHANGE NOTHING. 

More computers 

Need a station for every participant - especially in a group like this. 

Make it longer. Allow participants with various computer skills the time to dialogue more with the 
technical process, review and understand better some of the graphs and results that are presented. 

Form and content are both important and neither can be accomplished well in a short time span. 

I wish we'd have settled on a project this morning and spent the afternoon brainstorming our action plan. 
Let's do this over 2 days next time. 

Please comment on what you think about groupware; enter applications at your organization that 
you think would benefit from using groupware. 

I can see how this technology could help a group reach consensus on an issue 
It was very efficient. I wish more than one person could vote 
on a computer. 

The sharing and brainstorming time was superb! The past half hour has been our most productive of all the 
time our group has been together. 

We were able to use our best skills and build on one another's work. 

Time efficient. Allows for total participation if enough computers. Reporting becomes facilitated because 
can quickly print and finalize. 

Use the information for faster decision-making 

The last part of our session seemed to be more productive when their is less talking and more in-putting. 

Such a time saving device. Hopefully more people and organizations will be exposed to this soon. 
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Its fun to see all the comments come up so quickly and see that your own are conveyed just as fast. 

This would be very valuable with board of directors, with health professionals in grant selection. I also 
could see it useful with town hall type meetings. 

I like the annonimity of the software. Easy to use. I think that my [name deleted] could use something like 
this for the long range plan which we are developing. The process has been much, much too long because 
of logistical problems. We could also use this in committees for deciding on various issues. Ditto on 
brainstorming! 

I'd love to use it at our [name deleted] meetings. 

In general I like the concept. 

I have used Lotus notes once to judge a national awards contest for the [name deleted]. 

I think that more individual group dynamics and computer skills must be equal to fully take advantage of 
the potential of this tool. Our organization already uses e-mail and chat functions for similar uses. The 
polling and organization capabilities are the most useful for a sense of completion and record keeping. 

My firm practices [name deleted] and conducts closings which could be accomplished in multiple locations 
easier than bringing everyone to one location at the same time. 

Meeting C 

Where did the process succeed? 

EVERYONE CAN PARTICIPATE 

It was an interesting and new approach which opens a different approach to group problem solving. 

It forces individuals to respond to each issue. 

Technology for the new century used in a classroom setting 
Improvement could be made with remote group members. 

Successful in that it forms a permanent record of the case discussed in class-for future review of concensus 
and your own input. 

Everyone gets a chance to participate! 

Everyone should have an opinion. 

I enjoyed the exercise and hope it is used more in the future for industry and community. 

Quick, succinct participation by all students. 

It is apparent that this application, in some format, will be a part of all of our future workplaces. This helps 
us to recognize the worggroup concept as 
a vital part of our future office places 

Where did it need improvement? 

Hard return on comments section to make it more readable. Comments after each question. 

Increase anonymity. Very easy to figure out who is answering what. This is not a problem in this particular 
group, however it may be an issue in more controversial situations. 

Format problems with the responses could either be fixed with software commands (perhaps global 
linewrapping) or with more familiarity on the part of the participants. 

Formating text to automatically wrap whan approaching the end of the screen. 

The system should format responses as they are submitted. It was annoying to have to search the page. The 
wider margins are for spreadsheet applications. 

Duplication in both typing and talking. With practice, less talking could be achieved. 

Typing text should automatic wrap 

Try using a Hypertext based format so that each step is clear to the student. Facilitator interaction was 
~ required too often. 

For Windows Literate persons, it was relatively easy however. 

A potentially very powerful tool for very complex issues! 

Ananimity helps with unforced, unpressured interaction. 


General comments about groupware. 


ENJOYED DOING IT BUT FOR OUR SMALL CLASS AND OUR GROUP DISCUSSIONS, 
PROBABLY WASN’T NECESSARY. HAD WE ALL ANSWERED AHEAD OF TIME, IT MAY HAVE 
MADE THE PROCESS GO BY QUICKER. I LIKED IT OVERALL. 

The process was good in communicating overall results. 

I like the ability to interact at my own pace. Intellectual capacity is not necessarily measured in speed of 
response. 

Interesting approach to teaching 

Interesting and could be a fun way to hold a meeting. 

The technology is exciting. It is very useful for various situations. 

For case review this software has pluses and minuses compared to verbal presentation in class. I can 
imagine other uses in which the pluses would have greatly outweighed the minuses, e.g. team building. 


Meeting D 

Where did the process succeed? 

Gained input from a varied group of women. 

Anynimity 

The most user friendly tool I have used. 

It was nice to have my opinions heard and counted. 

Great for group concensus. I guess there is a way to take this phase to the next level of discussion. I would 
like to try that some day. 

It was fun. I think it would be difficult to do this with a lot of people, but the more input, the more fun the 
process can be. 

Gained input from a varied group of women. 

Anynimity 

The most user friendly tool I have used. 

It was nice to have my opinions heard and counted. 

Great for group concensus. I guess there is a way to take this phase to the next level of discussion. I would 
like to try that some day. 

It was fun. I think it would be difficult to do this with a lot of people, but the more input, the more fun the 
process can be. 

Where did the process need improvement? 

I see potential but until the server is faster the process is too slow. 

General comments about groupware 
The most user friendly tool I have used. 

Great tool! Especially for brainstorming if you have longer. 

Very user friendly! 

Love it. The wave of the future, I think. Makes surveying less painful.. .especially the data analysis. 

Remote access please. Didn't know there was a specific brainstorm tool. 

Meeting E 

From a software perspective, what did you like about the session? 

Anonymity, graphics, instant feedback- when working ...properly 

Very cool. Anonymity. 

ditto 

anonymity and instant feedback 

anonymity, rapid polling (when properly functioning) 

It makes the discussion more fun to work on a computer. I also liked the anonymity, 
neat software 

Anonymity of inputs made it easy to provide inputs. 



It was exciting to brainstorm with others at the same time, same place. And I liked the anonymity. 

This is an excellent system for use in an organization. The employee having access to the system at their 
leisure is good. 

From a software perspective, do you think any changes should be made, if so how? 

Maybe some sound effects... 
get the bugs worked out... 
improve the scoring techniques, 
no glitches in voting 

there are still some bugs in the system; I would concentrate on getting those out. 
voting 

Things were a little slow when we had to wait for the facilitator to set up the voting section. Surely this can 
be done faster? 

The fact that we had some participants who could not figure out the difference between entering comments 
and new ideas suggests that there is some explanation that is lacking. 

Not sure, but I believe the rating scale was not helpful. I would eliminate the final 5 ratings when drawing a 
conclusion. 

I was entirely unimpressed with the voting. There is more calling for raw scores in the kind of information 
we were using. 

General Comments. 

I really enjoyed using the software.. .it's too bad it wasn’t working properly this evening, 
it was neat-o 

could be effective if used properly. 

It was fun 

it worked better the first time I used this program; bugs need to be fixed 
I enjoyed this 

Excellent tool for easy exchange of ideas. 

Very enlightening and "futuristic!" 

Good learning tool 
Enjoyed it! 

There is too much dependence on the technical facilitator. Possibly when used on a long term basis, the 
people will know better how to use the system (which will help it to more effectively gather information). 


Meeting F 

From a software perspective, where did the process succeed? 

A variety of ideas 

In providing the means by which people can build on ideas typed in by other members 

I like it, it's like being in a chat room with metrics 

Fostered the organization of ideas in a uniform manner 

User friendly (except for [name deleted]) 

allowing people to voice their opinions anonymously 

less distractions 

it enhanced our typing skills 

A sharing of ideas 

promotes various ideas from people(brainstorming) 

~ From a software perspective, where did the process fail? 

None that I saw 

make the exercise focus on creativity and brainstorming 
A little slow at times 

Gave an opportunity to those people who were not taking this program seriously, a chance to clown around 
I didn't see any problems 
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the little black dots didn't always correspond with the terminal that created the comment 
need better terminals, screen is very bad 

Please provide any general comments about Groupware. 

good idea, we should assemble a group af students to work on ideas for the university 
understanding others in a more, relaxed way. 

this would be good to use to do our group evaluations after the presentations. 

I wish we had more time to bring in the graphical aspect, 
excellent approach to getting numerous ideas 
i want one for home 

we like it! we love it! we want some more of it! 

it helps get into the mind-set of others 

It would be very helpful to businesses as a whole 

these demonstrations should be done in smaller groups of people because these screens are too small and 
too blurry. 

Writing too small & not clear 
makes communication easier 
rock and roll 

advantage communication 
makes life so much easier 


Meeting G 

Where did the process succeed? 

Faster and more flexible than the old system 
Obviously, the anonymous input is valuable. 

Got a lot of ideas quickly 

I think it is a great piece of software. If a group uses it a number of times it probably could become a very 
useful analytical tool. 

It was a good tool for helping a group reach some sort of concensus/discussion points. 

allowing revision of ideas 

adding comments to buckets, great approach 

Simple to learn. 

Voting options a strong feature 

Where did it need improvement? 

I don’t think it is possible to really be effective with a software that is so easy that anyone can use it 
The system was slow and cumbersome. It should operate on its own server. 

Some sort of audible alarm that sounds when the time for any given segment is over. I noticed that no one 
noticed that the 10 minute time limit had expired during one of the periods when we were busy typing away! 
The system was a bit tardy at times. Things took longer than they probably should 
More diverse representation would have been helpful, 
system slow, broke train of activity 

Would help if we could easily bring up (from a menu) the questions we are addressing. 

General comments about Groupware 

This was a very effective way to collect information in a timely manner. 

It kept the project focused, cutting down on time consuming debates, and freeing discussion. 

I think the potential for this type of decision making and analysis in the future is absolutely enormous 
Nice format. Needs continuing development. 

Much improved over the DOS version but still have room to improve in speed and managerial efficiency. 
Seemed to cover most of the options I wanted 


Meeting H 



From a Software perspective, where did the process succeed 

Resulted in the generation of substantial number of ideas 

allowed great brainstroming 

great software ~ very smooth and capable 

wonderful demonstration of software-thank you 

everythng worked - for a change! 

seemed to work just fine 

It seemed to work fine, no problems 

brainstorming put ito practice & made painless 

software worked -pleasant surprise 


From a software perspective, where did the process need improvement? 

cant think of a thing, the bucket system works great 

The ability to colapse ideas into broader categories that can be ranked 

is there any way to group similar comments together electronically 

Can the program sort my key words 

This time no glitches — but notice that we resorted to consolidating items via the white board. Why 
couldn't we do the same via software? The 2-3 times I've led brainstorming and tried to consolidate via 
the computer system it was a mess. 

General comments about Groupware. 

it's seems awfully drab and dark in this room; is there any way to make it a little lighter in here and still be 

able to see the screen at the front? 

well done, very user friendly 

This is well suited to our program needs 

is there a Mac version 

Generally a good idea, although I am not convinced it adds much to the meeting that couldn't be done with a 
flip chart and Nominal Group Technique 
I need to be familiar with other packages to comment further 
advertise this to our local companies 

I see potential but until the server is faster the process is too slow. 

I think [name deleted] pre-work and leadership was essential for this working as well as it did 

Meeting I 

From a software perspective, what did you like about the session? 

The range of ideas generated. 

Much more efficient and organized than the verbal free flow of ideas 
The software was very easy to work with. 

Rapid brainstorming. Rapid polling of consensus. 

Being able to scroll over the range of ideas. 

Ease of Use 

Ability to simultaneously view other inputs and add your own ideas 
I liked the objectivity and efficiency of the medium. Very friendly and attractive software. 

Went well 

From a Software perspective, do you think any changes should be made, if so how? 

Concept merger is too cumbersome and facilitator centric. 

Get a faster server. 

Get the pentium processor, too much down time 

The dynamic renumbering at various points creates confusion. 

speed things up 

Decentralize control. Allow users to be doing one thing while facilitator does another. 

Faster server. 



Mode transition is too slow and invasive to the flow of activity. 

Need faster, easier method for synthesizing idea generated during brainstorming that is not facilitator driven 
A more efficient way to regroup. Software should enable people to enter groupings on their 
machine and then let system automatically reclassify those where there is consensus. 

General Comments 
I enjoyed the session. 

Went fairly well; if system were faster it would have gone a little more smoothly; a little continuity was lost 
due to slowness; results seem useful. 

This was a good session, and generated much information 

I enjoyed the experience very much. It removes the emtotionalism of face to face arguments. It also allows 
for people to be candid and direct. 

Comfortable, easy to express ideas 

The session was useful. It takes time for a group to learn how to work together in this modality. 

we should have done this earlier, and we should do it more often. Thanks for a wonderful session, [name 

deleted] 

Meeting J 

From a software perspective, where did the process succeed? 

Ease of use 

ease and quickness of compiling data 

I may be tired but the writing beside the bucket is somewhat out of focus, 
ability to compile information quickly and anonymously for discussion 
quick 

Rapid correction/editing capabilities 

in giving the opportunity to express thoughts in an easy way 

Ease of use, clear visual to use for editing thoughts, excellent process sequence 

bigger icons for late night users 

information quickly and easily projected on screen 

can instantly see everyone's responses 

From a software perspective, where would you make improvements? 
larger letters 

seemed fine from this limited use 

Expand screen where comments appear fully across screen 

Lack of use of computers, do not feel qualified to suggest improvements 

screen was hard for me to read 

Allow visuals to show individual remarks vs group remarks 

Allow comparative screens to show voting results vs original rankingEase of use 

General comments. 

I think the process and the software worked very well. Keep using it. 
great tool. Application opportnities are tremendous, 
very enjoyable exercise, ease of idea development and concensus 
program is very easy to use for succesful group dynamics 
most helpful exercise 

enjoyed using the equipment and found it very easy to adapt to 

Meeting J 

From a software perspective, where did the process succeed? 

In every category. 

This software seems to be very interactive and thought provoking, 
putting it all together/ menu format most helpful 
At this time I personally can't see the need for changes 


Allowing the concurrent entry of ideas along with anonymity. 

I think the software is very well thought out and user friendly. 

This was a very worth while exercise and should work well in other topic areas 
helps promote teamwork 

From a software perspective, where would you make improvements? 

Allowing more than one person to edit at a time. 

Don’t see a need for changes. 

instructions need to be clearer/ allow for corrections 

General comments. 

Good Process to bring diverse group together 

Very informative, enjoyed the class, it’s late, good night! 

This is a great program and we appreciate the oppurtunity you provided us. Thank you!! ! 

This process is not as interactive when comparing it to a non-computer based program, interaction between 
participants is limited because the computer is in the way. We need more communication as a team not 
with a computer 
promotes teamwork 

helpful/ good way to do this sort of thing. 

Excellent way to spend the evening! Thank you. 

Meeting K 

From a software perspective, where did the process succeed? 

Easy to understand and operate. 

Allowed each participant to work simultaneously and anonymously 

overall a good program 

expedited information from all participants 

My bucket didn't tip when i clicked on it as the instructions stated. Otherwise good 
Excellent in getting input from everyone before starting the discussion 
i believe you can never be too user friendly so keep trying. 

The information was usefull in keeping everyone on the same task (page) 

the program promoted free expression of ideas not because of ananymity but because writing has always 
facilitated freedom of thought and expression. 

From a software perspective, where would you make improvements? 

Software was user friendly, can't identify any improvements at this time 

You're asking me? I am not qualified to answer this question with a "qualified” answer! 

Can't think of any 

The only improvements could be in the area of the add Idea screen. If you double click on the bucket you 
lose the add idea screen. If a get could be installed that would automatically bring up the add idea screen 
then that would be a benifit to the novice user. 

General comments. 

I think this is an excellent tool for [name deleted] to us this in their goal setting workshops for the budget 
year 

The ranking process is helpful to see where we stand as a group 
Excellent tool for facilitating dialogue without unnecessary confrontations 

This is an excellent tool that could be used to take on the "road". By this to take to [name deleted] in order 
to help them develop future directions for their [name deleted] 

Easy to use and understand 

Good session! Many good ideas surfaced. 

This S/W should be marketed to the [name deleted] 

Bucket tipping was an excellent visual aide. 

This was an enlightening experience. 



Meeting L 

From a Software perspective, where did the process succeed? 

Ability to have input from all members which was anonymous was very helpful 
Allowed open communication between participants 
Ideas were discussed fairly 
No suggestions. 

Excellent way to communicate in a group setting. 

Interesting to see that the group as a whole realizes the same problems, 
able to see our issues are the same 
Encouraged us to commucation. 

From a software perspective, where did the process fail? 

Software does not allow to uncombine ideas that have been merged 
No suggestions. 

Instructions took too long 

Introduction should be kept brief, and workshop promptly started. 

Did not fail 

Please provide any general comments about Groupware. 

I think this is a great tool for problem-solving issues within a company. 

I think the software is very beneficial, 
this seems to be a positive tool. 

Great format and ideas that will be very beneficial for future use 
The session was very informative 
The sessions was informative. 

Wonderful 

This was a great idea 

Meeting M 

From a software perspective, where did the process succeed? 

I was overall satisfied with the software used in the process. 

Displaying others comments. 

This process will only succeed if the comments that we made are read, considered and acted upon. If they 
are not considered seriously, then this whole process is a failure and a waste of time. 

The software enabled everyone to have an active voice. 

Recording comments for future reference. 

Who will see comments? 

This is a good way to share multiple ideas simultaneously. 

Yes! 

Is that really what you think is going on? 

We did not have to listen to educators on their soap box 
YOU MUST BE A FACULTY MEMBER 
it is not polite to yell 

From a software perspective, where did the process fail? 

The software seemed adequate. Not qualified to speculate with regard to this question any further. 

Lost the human interaction normally experienced when discusssing issues face to face 
Yes, but many more ideas can be shared in a lot less time this way. 

It does not seem to have any particular problems from my perspective. 

No methods to set a time limit for each question. To answer someone’s comment, the procedure is not 
automatic enough. 
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I find this a somewhat strange way to communicate. 


Please provide any general comments about Groupware. 

Users should be separated/partitioned. 

Quite good program but needs some improvements. 

The person using this lab should be made aware of the fact that they are particpating in someone’s 
"scholarly endeavor". They should also be given the opportunity( several weeks in advacne) to read any 
published literature which the organizers of this experiment have used to design, prioritize and evaluate the 
questions posed to the participants. The human subjects committee should have knowledge of these groups. 

Meeting N 

From a software perspective, where did the process succeed? 

This is my first time on this type of proceedure. I find it facinating and extremely easy to use. 

Gathers a lot of information 

Same here. 

An easy way to gather comments. 

From a software perspective, where did the process fail? 

I really don't have enough experience to be able to answer this question acturately. 

Highly redundant process with information overload. 

I think the redundancy was due to the design of the questions, not the software. 

Takes too much time. I don't see the real advantage of an oral exchange other than a real time record. 

We did not really take advantage of all the features of this software. 

Did not see enough of the system to properly respond to this question. 

Please provide any general comments about Groupware. 

I seems to make it easier to state how you feel about a subject or problem without the fear of retaliation. 

The real improvement would be able to reduce the reduncy as the process proceeds. 

I have not really seen the power of this software from this exercise. 

Need to learn more with other aspects of the system to comment on this system. 

Meeting O 

From a software perspective, where did the process succeed? 

Got things out in the open. And in writing (kind of). Anonymity (kind of) might have been a plus. But I 
still think a discussion face-to-face would have been better. 

I liked it. On such a sensitive topic, face-to-face interaction may have caused some problems. This way, 
people could sya what they wanted to without fear. It was fun too. 

I agree - a discussion would have been more productive and FAR faster. I resent the amount of time this 
has taken. Have we really said enough that’s new to justify the amount of [name deleted Rime expended on 
this? 

Anonymity has its benefits and its problems. For the most part, it succeeded here in allowing people to 
speak freely. I liked seeing responses to what I had written and not worrying that others would judge me 
based on those comments. 

I got to hear comments from others which were similar to mine, when I thought I was the only one who 
thought this way. 

i liked the anonymity, but i distrust it. if i can't say what i need to say without being anonymous, i suspect 
something is wrong with what i have to say. i think people should be encouraged 

1 . to take responsibility for their opinions, and 

2. take responsibility for communicating them to others in a decent way. 
both these things are necessary. I bet you are tenured! 
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OOOPS. i did not answer the question, common problem for me. 

This way, nontenured and tenured faculty were able to comment freely. On topics like these nontenured 
faculty are left out and for good reason, they are afraid to speak their minds for fear of being judged by 
more powerful colleagues. 

The bigger the group, the better. 

Ditto all of the above: I liked the anonymity, but I would not have minded saying what is said here in open 
face-face forum. I am opinonated and do not care if I offend people or not. I do thing [name deleted] has a 
bright future in research if some things change. 


From a software perspective, where did the process fail? 

Difficult, for this beginner, to follow other people's comments. Lacks the quickness of a good conversation. 
The Lock feature (by which only one person at a time can respond to a posting) is no good. 

I enjoyed it. 

Need a way to indicate additions. Some sign in the margin to show that there's been something added. I 
wasted too much time reading over material looking for the comments. 

Changing this would be a mojor improvement 

This is not exactly a failure, but it is difficult to see when a comment has been added. All you need is a 
notation in the comments section each time someone adds something. 

Also, the screen flashes and scrolls when a comment is added. This is very disturbing to the eye and made 
me lose my place several times. Perhaps another method of indicating that a comment has been submitted. 
If there was a way that we could it on our own time with out having to be tied down to certain time 
schedule, this would be great. In fact, I really like. 

difficult to figure out what was added, might highlight the material being added. 

difficult to scroll back and forth and find thing. 

It was OK. I thought the process was enjoyable even fun at times. 

a little too openended? i have little sense of accomplishment, but maybe that's okay, it did seem, however, 
somewhat like a bad meeting, it did not feel like good conversation, where there is a back and forth and 
where you can watch the other person, or persons, and where the sharing is not simultaneous, the 
simultaneity and anonymity of comment is sort of counterproductive. 

Please provide any general comments about Groupware. 

Can it be used for writing classes? That is, can an essay be loaded up and can students use the chat box to 
make commets? 

I would do this again. I know others may not, but I thought this was helpful. 

Great start ! 

Might be nice to be provided with a summary of what went on 
I liked it too. I bet all the "likers" are untenured! 

Now why would the "likers" be untenured. Because we are more open minded. 

I would also do this again. I hope the software can help with the overall problem at [name deleted] 
regarding research. After all, I came to this meeting with the expectations of airing my views on the future 
of research at [name deleted]. Someone had better read this stuff. 


Xll 


Analysis of Session Feedback Collected at Meetings 
June 1995 - May 1996 at RICIS 

Fifty seven meetings were held in the Group Support Systems Research Laboratory at RICIS. These 
sessions can be broadly divided into two segments: 

1. External Groups (from business, government and proffesional organizations) 

2. University of Houston - Clear Lake: 

The School of Business, 

Other groups within the university. 


External Groups 

There were nine meetings held during the period between June 1995 and May 31, 1996. The session 
feedback results convey an of overall apreciation of the groupware process . Other features that participants 
thought were advantageous were, the ability to build on one anothers’ work, the ability to make the decision 
making process faster, annonymity and ease of use of the system. There were requests to install more 
terminals within the facility. 

School of Business 

The School of Business used the GSS lab in their classes and partially funded a student to assist with the 
sessions. 

Twenty four meetings were held during the period specified above. 

Advantages participants identified included: the ability to engage in different approaches to group problem 
solving; participation extending to every participant; avoiding the meeting being dominated to a few 
members; allowing for a permanent record of all meeting data; the ability to collapse ideas into broader 
categories that can be ranked;.and, stressed repeatedly, the annonymous feature of the system. 

Disadvantages identified included: the network being slow; too much dependance on the technical 
facilitator; .and the potential for lack of annonymity when typing in comments, i.e., that the person sitting 
next to them could see what they typed in. 

Other groups within the University 

There were twenty four.meetings. A University wide survey regarding the future of academic activities of 
UHCL was conducted over a period of three months. Training in the use of Group Systems for Windows 
was provided for employees of UHCL. Both these activities were carried out through a Total Quality 
Management (TQM) grant. 

Some participants compared the windows version that was used, with the DOS version and expressed the 
view that the windows version of Group Systems was a better, improved version. They felt it was more user 
friendly and allowed for open communication between participants. Overall these participants felt that felt 
that the voting options, the annonymous feature of the system and ability of rapid polling were definite 
advantages of the system. 

Dealing with disadvantages, the participants felt that the system was slow, and a solution for this problem 
was put forward to obtain a faster server.. Some participants felt that the use of the system limited human 
interaction. One comment by a user expressed the idea of using partitions to separate the participant 
terminals and enhance the annonymous feature of this system. 



Analysis of the GroupSystems for Windows tools used: 

Tool Number of times used 

Vote 62 

Categorizer 55 

Topic Commenter 47 

Survey 18 

Group Outliner 4 

Electronic Brainstorming 3 
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